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Abstract 

The  primary  objective  of  this  thesis  was  to  compare 
three  knowledge  acquisition  techniques  used  to  gather 
knowledge  for  the  development  of  an  expert  system.  The  goal 
was  to  determine  which  technique  produced  knowledge  in  a 
form  most  suitable  for  incorporation  into  an  expert  system. 

The  three  acquisition  techniques  compared  were 
interviewing,  task  observation,  and  concept  mapping.  Three 
experts  were  selected  and  randomly  paired  with  a  technique. 
Knowledge  acquisition  sessions  were  then  conducted  with  each 
expert  using  the  technique  assigned  to  that  expert.  The 
knowledge  extracted  from  these  acquisition  sessions  was  then 
compared. 

Overall,  concept  mapping  produced  more  rules,  in  less 
time,  and  with  fewer  inferences  than  the  interview  or  task 
observation  techniques.  Additionally,  the  knowledge  base 
acquired  through  the  concept  mapping  technique  was  more 
complete.  Finally,  concept  mapping  required  one  less 
translation  of  the  knowledge  to  arrive  at  a  form  necessary 
for  programming  the  expert  system. 

An  expert  system  was  developed  using  the  concept 
mapping  technique  and  was  validated  in  a  field  test. 

Results  showed  that  the  solutions  provided  by  the  expert 
system  matched  those  provided  by  the  human  experts .  /  / 
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A  COMPARISON  OF  KNOWLEDGE  ACQUISITION  TECHNIQUES 
USED  IN  THE  DEVELOPMENT  OF  EXPERT  SYSTEMS 


I.  Introduction 


Background 

An  expert  system  is  "an  artificially  intelligent 
computer  program  that  solves  problems  at  an  expert  level  by 
using  the  knowledge  and  problem  solving  logic  of  human 
experts"  (9:68).  The  popularity  of  expert  systems  as  tools 
in  decision  making  is  growing  rapidly  in  business  and 
industry  (25:152).  Corporations  are  committing  significant 
resources  to  the  development  and  application  of  expert 
systems  in  order  to  achieve  improvements  in  operational 
effectiveness  and  efficiency  (9:68).  In  the  Air  Force, 
expert  systems  are  being  developed  to  solve  a  variety  of 
problems  in  areas  such  as  aircraft  maintenance,  acquisition, 
supply,  and  civil  engineering. 

One  of  the  most  difficult  and  time-consuining  activities 
in  the  development  of  these  expert  systems  is  the  process  of 
knowledge  acquisition  (20:269).  Kim  and  Cou-tney  define 
knowledge  acquisition  as  "the  process  of  gathering  knowledge 
about  a  domain,  usually  from  an  expert,  and  incorporating  it 
into  a  computer  program"  (20:269).  While  this  knowledge  can 
come  from  a  number  of  different  sources  such  as  text  books, 
journals,  and  data  bases,  the  emphasis  in  knowledge 
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acquisition  continues  to  be  placed  on  the  human  expert 
(19:53).  Because  of  the  time  and  difficulty  associated  with 
extracting  knowledge  from  the  human  expert,  knowledge 
acquisition  has  been  cited  throughout  the  literature  as  a 
critical  "bottleneck"  in  the  development  of  expert  systems 
(4:144;  5:228;  20:269;  25:152). 

Despite  continued  advances  in  expert  system  technology, 
the  process  of  knowledge  acquisition  remains  ill-defined. 
There  is  much  written  in  the  literature  on  how  to  build 
expert  systems,  but  little  written  about  knowledge 
acquisition  (7:152).  Even  books  that  describe  in  detail  how 
to  build  an  expert  system  do  not  say  much  about  the  subject 
of  knowledge  acquisition  (7:152).  This  lack  of  direction 
can  have  a  negative  impact  on  the  successful  development  of 
expert  systems  the  Air  Force. 

Problem  Statement 

The  power  of  an  expert  system  is  directly  dependent  on 
the  quality  and  completeness  of  its  knowledge  base  (20:269). 
The  acquisition  of  this  knowledge  base  is  the  problem  that 
most  limits  the  application  of  expert  system  technology. 

Many  acquisition  cechniques  have  been  identified  in  the 
literature,  but  very  little  data  is  available  on  which 
technique,  or  combination  of  techniques,  may  be  more 
effective  or  more  useful  given  different  situations. 
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Research  Objective 


The  purpose  of  this  research  was  to  evaluate  and 
compare  the  effectiveness  of  three  knowledge  acquisition 
techniques  used  to  gather  knowledge  from  an  Air  Terminal 
Operations  Center  (ATOC)  in  an  Air  Force  Transportation 
Squadron.  The  goal  was  to  determine  which  of  those 
tecniques  produced  knowledge  in  a  form  most  suitable  for 
incorporation  into  an  expert  system. 

Research  Questions 

The  following  questions  were  answered  to  solve  the 
specific  problem. 

1.  What  are  the  current  and  most  widely 
recognized  techniques  used  to  acquire  knowledge  from 
experts? 

2.  What  current  Air  Force  issue  in  the  ATOC  was 
appropriate  for  building  a  prototype  expert  system  and  what 
software  development  tool  could  be  used  to  build  it? 

3.  Given  a  selected  application  and  software 
development  tool,  which  extraction  technique  among  three 
evaluated  was  more  effective  in  producing  the  knowledge 
which  would  be  programmed  into  a  prototype  expert  system? 

4.  After  developing  a  prototype  expert  system 
using  one  of  the  three  predetermined  acquisition  techniques, 
was  the  knowledge  base  complete  and  the  prototype  system 
valid? 
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Scope  of  the  Study 


Although  there  are  several  knowledge  acquisition 
techniques,  this  research  compared  only  three.  The  expert 
system  developed  from  the  knowledge  acquisition  process  was 
a  prototype  system  focusing  on  a  specific  problem  related  to 
the  activities  of  the  Air  Terminal  Operation  Center  in  an 
Air  Force  Transportation  Squadron. 

Method  of  Organization 

Chapter  I  introduces  the  concept  of  expert  system 
technology  and  provides  background  informat  i  on  the 
process  of  knowledge  acquisition.  This  chapter  also  defines 
the  specific  problem,  research  objective,  research 
questions,  and  scope  of  the  study.  Chapter  II  discusses 
background  information  on  the  development  of  expert  systems 
and  the  process  of  knowledge  acquisition.  Chapter  III 
presents  the  specific  methodology  that  was  followed  to 
compare  different  knowledge  acquisition  techniques  and  to 
evaluate  their  effectiveness.  Chapter  IV  reviews  the 
findings  and  analyzes  the  results  of  the  research  effort. 
Chapter  V  presents  the  conclusions  and  recommendations. 
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II.  Literature  Review 


Overview 

The  purpose  of  this  chapter  is  to  review  some  of  the 
literature  written  about  expert  systems  and  the  knowledge 
acquisition  methods  used  in  their  development.  This  chapter 
is  divided  into  three  main  parts.  The  first  part  discusses 
expert  systems  in  general  by  defining  an  expert  system  and 
providing  a  description  of  its  major  components.  Next,  the 
steps  required  in  the  development  of  expert  systems  and  some 
of  the  programming  tools  used  to  build  them  are  presented. 
Finally,  some  of  the  benefits  and  limitations  of  experts 
systems  are  discussed.  The  next  major  part  in  the  review 
deals  with  the  issue  of  knowledge  acquisition.  The 
importance  of  knowledge  acquisition  is  discussed  and  some  of 
the  difficulties  associated  with  the  process  are  presented. 
Then,  some  of  the  methods  for  knowledge  acquisition  are 
introduced.  The  final  section  in  this  chapter  addresses  the 
appropriateness  of  applying  expert  system  technology  in  an 
Air  Terminal  Operations  Center  (ATOC). 

Background  on  Expert  System 

Definition  of  an  Expert  Systems .  There  are  many 
definitions  for  experts  systems  offered  in  the  literature. 
Saylor  defines  an  expert  system  as  "a  problem-solving  or 
decision  making  computer  program  designed  to  perform  the 
same  evaluation  as  a  human  expert  in  a  specified,  defined 
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area"  (27:450).  Carpenter  describes  an  expert  system  as  "a 
form  of  computer  software  that  relies  on  stored  facts  and 
rules  of  thumbs  to  mimic  the  decision  making  of  human 
experts"  (6:64).  Cook  defines  an  expert  systems  as  "an 
artificially  intelligent  computer  program  that  solves 
problems  at  an  expert  level  by  utilizing  the  knowledge  and 
problem  solving  logic  of  human  experts"  (9:68). 

Expert  systems  differ  significantly  from  conventional 
computing  systems.  While  conventional  computer  programs 
deal  primarily  with  quantitative  data  and  are  based  on 
clearly  defined,  step-by-step  procedures,  expert  systems 
deal  primarily  with  qualitative  information  and  are  able  to 
reason  about  data  and  draw  conclusions  employing  heuristic 
rules  (28:17).  Heuristics  are  sometimes  characterized  as 
the  "rules  of  thumb"  that  one  acquires  through  practical 
experience  to  solve  everyday  problems  (28:17). 

Components  of  an  Expert  System.  Every  expert  system 
consists  of  three  major  components:  a  knowledge  base,  an 
inference  engine,  and  some  form  of  user  interface.  The 
heart  of  the  expert  system  is  its  knowledge  base.  The 
knowledge  base  contains  the  facts,  rules,  and  other 
knowledge  required  to  solve  a  problem  (32:8).  The  inference 
engine  works  on  the  knowledge  base  to  reach  conclusions 
about  the  problem  (1:18).  The  inference  engine  can  work 
forward  through  a  problem  from  an  initial  set  of  conditions 
to  a  specific  goal  (13:87-88).  This  is  called  forward 
chaining.  The  inference  engine  can  also  work  backward  from 
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a  goal  to  a  set  of  premises  that  support  that  specific  goal 
(13:87-88).  This  is  referred  to  as  backward  chaining.  Some 
expert  systems  incorporate  both  forward  and  backward 
chaining  strategies  (13:87-88).  The  user  interface  is  the 
part  of  the  program  that  allows  the  user  to  communicate  with 
the  system.  The  user  interface  will  ask  questions,  present 
menu  driven  choices,  and  communicate  to  the  user  the  answer 
or  solution  once  it  has  been  found  (32:18).  Figure  1 
illustrates  the  basic  structure  of  an  expert  system. 


Other 

Database 

Interfaces 


Facts 

and 

Data 


A 


Dynamic 

Working 

Memory 


Facts 

and 

Data 


Advice 

and 

Solutions 


▼ 


▼ 


Knowledge  Base 

_ i _ 

Inference  Engine 

Facts 

Control  Strategies 

Heuristic  Rules 

Frames 

Figure  1.  Basic  Structure  of  an  Expert  System  (2:27) 
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Developing  an  Expert  System.  The  development  of  expert 
systems  is  an  iterative  process,  as  illustrated  in  Figure  2 
(2:107) . 


Systems 

Requirements 

Analysis 


General  Problem 
Definition 


Knowledge 

Engineering 


Prototype 

Development 


Verification  and 
Val idation 


Figure  2.  Expert  System  Development  Process  (2:107) 

Analyze  System  Requirements .  The  first  step  in  the 
development  process  is  to  determine  the  user's  needs  and 
develop  an  understanding  of  the  problem  area  to  be  addressed 
by  the  system  (2:106). 

General  Problem  Definition .  Because  of  the 
uncertain  and  qualitative  nature  of  the  problems  typically 
addressed  by  expert  systems,  the  problem  is  first  defined  in 
very  general  terms  (2:106-107).  A  more  in-depth  definition 
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of  the  problem  is  then  developed  through  a  series  of 
knowledge  engineering  sessions  with  the  human  expert 
(2:107)  . 

Knowledge  Engineering.  Knowledge  engineering  can 
be  defined  as  the  process  of  acquiring  the  knowledge  from 
the  human  expert  and  then  translating  that  knowledge  to  a 
form  that  can  be  used  in  the  search  for  and  generation  of 
solutions  to  problems  (29:357).  It  is  also  an  interative 
process  in  which  many  meetings  with  the  expert  may  be 
required  to  gather  all  the  pertinent  data  needed  for  the 
knowledge  base  (21:8). 

Prototype  Development .  After  one  or  two  knowledge 
acquisition  sessions  with  the  expert,  a  working  prototype 
system  should  be  developed  (2:107).  This  prototype  system 
can  then  be  used  in  subsequent  knowledge  acquisition 
sessions  to  determine  if  there  is  any  missing  knowledge  or 
if  any  modifications  are  required  (2:107). 

Verification  and  Validation .  With  each  session, 
the  prototype  is  refined  and  enhanced.  The  system  is 
verified  for  consistency  and  completeness  (22:70). 

Redundant,  conflicting,  circular,  and  unnecessary  conditions 
and  rules  are  either  changed  or  removed  (22:70).  Eventually 
a  complete  system  will  emerge.  One  final  step  at  this  point 
is  to  validate  the  expert  system's  overall  value  and  to 
answer  the  question  of  just  how  closely  the  system  performs 
to  human  expert  levels  (24:91). 
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Development  Tools  (Software) .  When  the  decision  is 
made  to  develop  an  expert  system,  a  software  development 
tool  will  be  required  to  help  create  the  system.  Expert 
systems  can  be  created  using  any  of  the  common  programming 
languages,  such  as  BASIC,  Fortran,  C,  Pascal,  Forth,  PROLOG, 
LISP,  and  Assembly  Language  (14:624).  Programming  languages 
can  address  highly  complex  problems  and  provide  a  great  deal 
of  flexibility  (2:120).  However,  the  use  of  programming 
languages  can  be  a  complex  process  which  may  involve  special 
programmers,  special  operating  environments,  extensive 
hardware,  extended  development  time,  and  a  great  deal  of 
money  (14:824).  An  alternative  to  programming  languages  is 
expert  system  shells.  Expert  system  shells  can  address  a 
wide  range  of  problems  and  can  be  very  inexpensive  and  easy 
to  program  (29:9). 

Expert  System  Shel Is .  Expert  system  shells  are 
special  software  packages  created  specifically  to  help  build 
expert  systems.  They  are  similar  in  some  respects  to 
conventional  software  packages  like  data  base  management 
systems  or  spreadsheets.  A  shell  provides  the  basic 
framework  in  which  data  or  knowledge  can  be  entered  and 
manipulated  in  predefined  ways  (14:824). 

An  expert  system  shell  does  not  contain  a  knowledge 
base.  The  major  distinguishing  difference  among  expert 
systems  built  using  expert  system  shells  is  the  content  of 
the  knowledge  base.  The  inference  engine,  data  base,  and 
user  interface  will  work  with  many  different  knowledge 
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bases.  Knowledge  is  simply  coded  in  the  designated  format 
and  entered  into  the  knowledge  base  (16:45). 

Benefits  of  Expert  Systems .  Expert  systems  offer 
several  benefits.  They  can  preserve  the  crucial  expertise 
of  the  human  expert  that  could  otherwise  be  lost  (27:451). 
This  expertise  can  be  easily  transferred  by  simply  copying 
the  program  (27:451).  Expert  systems  are  less  expensive 
than  human  expertise,  less  likely  to  make  mistakes,  and  can 
be  easily  documented  (27:451).  Finally,  expert  systems  can 
free  human  experts  from  the  performance  of  routine  or  simple 
tasks  (27:451). 

Limitations  of  Expert  Systems .  Despite  these  benefits, 
there  are  limitations  to  expert  systems.  Expert  systems 
work  best  when  applied  to  narrowly  defined  tasks  (6:64-65). 
They  are  less  successful  when  dealing  with  larger,  more 
complex  problem  domains  (6:64-65).  Additionally,  expert 
systems  are  only  computer  programs,  and  their  decisions  can 
be  very  unimaginative  when  compared  to  the  decisions  of  a 
human  expert  (27:451).  The  biggest  limitation  of  expert 
systems,  however,  is  the  difficulty  of  extracting  the 
expertise  from  the  human  expert  (27:451). 

Knowledge  Acguisition 

Kim  and  Courtney  define  knowledge  acguisition  as  "the 
process  of  gathering  knowledge  about  a  domain,  usually  from 
an  expert,  and  incorporating  it  into  a  computer  program" 
(20:269).  The  power  of  any  expert  system  comes  from  the 
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knowledge  contained  in  its  knowledge  base  (20:269).  While 
this  knowledge  can  come  from  many  sources  such  as  text 
books,  journals,  and  data  bases,  the  emphasis  in  knowledge 
acquisition  has  been  placed  on  the  human  expert  (19:53). 
Because  of  the  time  and  difficulty  associated  with 
extracting  knowledge  from  the  human  expert,  knowledge 
acquisition  has  been  cited  throughout  the  literature  as  a 
critical  bottleneck  in  the  development  of  expert  systems 
(4:144;  5:226;  20:269;  25:152).  Knowledge  acquisition  is 
the  most  important  process  in  the  development  of  expert 
systems  (16:158). 

Difficulties  with  Knowledge  Acquisition.  The  process 
of  knowledge  acquisition  is  dependent  upon  the  expert's 
abilities,  availability,  and  willingness  to  cooperate 
(26:44).  There  are  several  difficulties  associated  with 
those  requirements.  Human  experts  have  a  difficult  time 
explaining  how  they  make  decisions  (27:451).  This  inability 
of  an  expert  to  verbalize  how  he  or  she  goes  about  solving  a 
problem  is  a  difficulty  in  knowledge  acquisition  cited  most 
often  in  the  literature  (4:144;  5:228;  20:269;  25:152).  The 
availability  of  the  expert  is  another  reason  why  knowledge 
acquisition  is  difficult  (5:228).  The  process  of  knowledge 
acquisition  takes  time  (5:228).  The  expert  or  the  expert's 
supervisors  may  not  be  willing  to  spend  the  time  required  to 
complete  the  project  (5:230).  The  willingness  of  the  expert 
to  participate  in  the  knowledge  engineering  project  is 
another  difficulty  in  knowledge  acquisition  (25:153).  An 
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unwilling  expert  will  cause  the  knowledge  acquisition 
process  to  fail  (25:153).  An  expert  may  be  unwilling  to 
participate  for  any  number  of  reasons  (25:153).  The  expert 
may  fear  his  or  her  job  may  be  eliminated  if  a  computer  can 
capture  their  expertise  (25:153).  The  expert  may  also  fear 
the  loss  of  esteem  others  hold  for  him  or  her  if  the  task  he 
or  she  performs  is  reduced  to  something  simple  through  the 
knowledge  acquisition  process  (25:153).  Finally,  the  expert 
may  not  know  how  to  explain  his  or  her  problem  solving 
expertise  and  may  fear  being  seen  has  inarticulate  (25:153). 

Methods  of  Knowledge  Acquisition.  According  to  Teft, 
expertise  consists  of  knowledge  accumulated  through 
experience.  With  the  possible  exception  of  those  who  teach, 
few  experts  spend  very  much  time  explaining  their  knowledge. 
To  elicit  knowledge  from  the  expert,  the  knowledge  engineer 
must  understand  the  ways  in  which  the  expert  relates 
objects,  relationships,  conditions,  constraints,  and  events 
within  the  area  of  expertise  and  then  apply  the  appropriate 
knowledge  acquisition  tool  (29:203). 

The  literature  discusses  several  methods  of  knowledge 
acquisition.  Olson  categorized  these  methods  into  direct 
and  indirect  methods  of  acquisition.  Direct  methods  focus 
on  the  explicit  knowledge,  knowledge  that  the  expert  can 
communicate  verbally.  The  direct  methods  include 
interviews,  questionnaires,  protocol  analysis,  interruption 
analysis,  task  observation,  drawing  closed  curves  and 
inferential  flow  analysis  (25:153-166).  Indirect  methods 
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focus  on  implicit  knowledge,  knowledge  that  cannot  be  easily 
verbalized.  They  collect  information  that  requires 
inferences  to  be  made  about  the  exact  nature  of  the  expert's 
knowledge.  The  indirect  methods  include  multidimensional 
scaling,  hierarchical  clustering,  general  weighted  networks, 
ordered  trees  from  recall  and  repertory  grid  analysis. 

This  list  of  direct  and  indirect  methods  is  not  all 
inclusive.  However,  it  should  give  the  reader  an  idea  of 
the  variety  of  techniques  available  to  the  knowledge 
engineer.  Heatherton  and  Vikan  provide  a  summary  of  each 
of  these  direct  and  indirect  methods  and  a  list  of  sources 
where  the  reader  can  find  additional  information  on  each 
technique  (17:32-46).  In  addition,  the  three  techniques 
selected  for  examination  in  this  research  project  are 
described  in  Chapter  III  and  Appendix  B. 

Air  Terminal  Operations  Center 

Expert  system  technology  is  being  applied  to  solve 
problems  in  a  variety  of  Air  Force  organizations  and 
activities  (8:1).  One  activity  in  which  expert  system 
technology  has  potential  usefulness  is  in  an  Air  Terminal 
Operations  Center  (ATOC).  The  ATOC  in  an  Air  Force  Aerial 
Port  Squadron  or  Transportation  Squadron  is  responsible  for 
coordinating  and  directing  all  transportation  activities  in 
support  of  airlift  operations  (11:5-1).  These  activities 
include  airlift  mission  setup,  load  planning,  fleet 
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services,  cargo  loading  and  unloading,  hazardous  cargo 
handling,  passenger  processing,  and  baggage  handling. 

The  ATOC  is  a  complex  operation,  requiring  close 
monitoring  of  many  different  events  which  must  be  completed 
in  sequence  and  on  time  to  ensure  the  success  of  the  airlift 
mission.  Besides  ensuring  the  successful  completion  of  the 
sequence  of  events,  personnel  in  the  ATOC  are  often  faced 
with  a  variety  of  problems  which  have  an  impact  on  the 
sequence.  Weather,  maintenance  problems,  and  traffic  delays 
are  only  a  few  of  the  variables  ATOC  personnel  must  deal 
with . 

Given  the  complexity  of  the  ATOC  operation,  reliance  on 
experienced  and  expert  personnel  in  the  positions  of  duty 
officers,  shift  supervisors  and  information  controllers  is 
commonplace.  Unfortunately,  ATOCs  are  often  unable  to 
retain  the  expertise  of  the  personnel  they  train  because  of 
rapid  turnover  due  to  PCS  moves,  separation,  retirement,  or 
in-house  promotions  to  different  positions. 

One  way  to  retain  the  expertise  of  the  experienced 
personnel  would  be  the  application  of  an  expert  system. 

This  expert  system  would  assist  the  inexperienced  person  by 
automatically  monitoring  the  status  of  the  transportation 
activities,  insuring  all  steps  in  checklists  were  completed, 
identifying  potential  problems  and  recommending  solutions  to 
those  problems. 
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Summary 


This  literature  review  provided  a  description  of  an 
expert  system  and  its  major  components.  It  presented  some 
of  the  advantages  of  expert  systems  and  some  of  the 
limitations.  It  pointed  out  the  importance  of  knowledge 
acquisition  in  the  development  of  an  expert  system  and 
identified  some  of  the  difficulties  associated  with  the 
process  of  knowledge  acquisition.  Some  of  the  methods  used 
to  extract  knowledge  from  the  human  expert  were  listed. 
Finally,  the  reasons  for  applying  expert  system  technology 
in  and  Air  Terminal  Operation!.  Center  were  discussed. 
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III.  Methodology 


Overview 

The  purpose  of  this  chapter  is  to  explain  the 
methodology  that  was  used  to  solve  the  research  problem.  In 
order  to  determine  which  knowledge  acquisition  technique 
among  three  selected  for  study  produced  more  effective 
results  when  compared  with  the  other  two  techniques,  several 
major  steps  were  required.  These  steps  consisted  of 
conducting  a  literature  review,  selecting  the  problem  area 
to  be  addressed  by  the  expert  system,  selecting  three  human 
experts,  selecting  an  expert  system  shell,  and  selecting  the 
three  knowledge  acquisition  techniques  to  use  in  the  study. 
After  these  steps  were  completed,  a  comparative  study  was 
made  in  which  the  selected  acquisition  techniques  were  used 
in  knowledge  acquisition  sessions  with  human  experts.  The 
results  of  these  sessions  were  then  compared  and  the  one 
technique  considered  more  effective  among  the  three  was 
selected  and  used  to  build  a  prototype  expert  system.  The 
accuracy  of  the  knowledge  base  in  the  prototype  expert 
system  was  then  validated  in  a  field  test  to  further  support 
the  effectiveness  of  the  selected  technique. 

Literature  Review 

The  initial  phase  of  the  study  consisted  of  conducting 
a  literature  review  to  examine  current  concepts  in  expert 
system  technology  and  knowledge  acquisition.  The  review 
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provided  a  definition  of  expert  systems,  a  description  of 
their  components  and  an  explanation  on  how  an  expert  system 
works.  It  presented  the  steps  which  are  necessary  to  build 
expert  systems  and  the  programming  tools  used  to  develop 
them.  The  review  also  examined  the  issue  of  knowledge 
acquisition  as  a  critical  process  in  the  development  of 
expert  systems.  It  identified  the  difficulties  and  problems 
related  to  knowledge  acquisition  and  listed  some  of  the 
methods  of  knowledge  acquisition.  Finally,  the  review 
provided  some  background  information  on  the  Air  Terminal 
Operations  Center. 

Selecting  the  Problem  Area 

The  Air  Terminal  Operations  Center  (ATOC)  in  the 
Transportation  Division  of  the  2750th  Logistics  Squadron, 
Wright-Patterson  Air  Force  Base,  Ohio  was  selected  as  the 
functional  area  in  which  the  expert  system  was  applied. 
Although  this  ATOC  is  not  the  largest  or  the  busiest  ATOC  in 
the  Air  Force,  it  possesses  the  characteristics  that  make 
the  application  of  expert  system  technology  appropriate. 

The  ATOC  is  an  area  in  which  valuable  human  expertise 
is  scarce  and  is  often  in  jeopardy  of  being  lost.  Many  of 
the  tasks  performed  are  complex,  unstructured  and  rely  on 
heuristic  solutions.  Heuristics  are  sometimes  characterized 
as  the  "rules  of  thumb"  that  one  acquires  through  practical 
experience  to  solve  everyday  problems  (28:17).  There  is 
also  substantial  variability  in  the  ability  of  assigned 


18 


personnel  to  perform  tasks.  At  one  end  of  the  spectrum  are 
the  newly  assigned  personnel  who  are  just  beginning  to  gain 
competence  through  experience.  At  the  other  end  are  the 
recognized  experts,  or  at  least  those  personnel  who  are 
recognized  as  being  the  most  experience.  It  is  not  always 
possible  to  get  a  good  mix  of  experienced/inexperienced 
personnel  on  each  shift.  An  expert  system  could  reduce  this 
variability. 

An  additional  reason  for  selecting  this  ATOC  is  that  it 
is  located  in  close  proximity  to  the  School  of  Systems  and 
Logistics.  This  eliminated  the  need  to  spend  TDY  funds  to 
conduct  the  knowledge  acquisition  sessions  and  made  the  data 
used  in  the  study  readily  assessable. 

The  Specific  Problem.  In  order  to  determine  one  or 
more  specific  problem  or  decision  areas  which  might  be 
effectively  addressed  by  an  expert  system,  an  interview  was 
conducted  with  Mr.  John  Altick,  the  Air  Terminal  Manager  at 
Wright  Patterson  AFB,  OH.  One  area  that  has  received  a 
great  deal  of  attention  recently  is  the  movement  of  space 
available  passengers. 

Extreme  care  must  be  taken  when  determining  the  number 
of  seats  available  for  use  by  space  available  passengers. 
Space  available  travel  is  considered  a  special  service 
benefit  and  offering  seats  to  space  available  travelers, 
only  to  have  to  cancel  them  because  of  some  mistake,  is  not 
only  a  matter  of  providing  poor  customer  service,  it  can 
also  be  damaging  to  morale.  The  number  of  variables  which 
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roust  be  considered  when  making  a  determination  on  the  number 
of  seats  to  release  is  large.  The  type  of  aircraft, 
destination,  number  of  seats,  service  facilities,  planned 
cargo  and  fuel  load,  crew  size,  and  crew  complement  are  just 
a  few  of  the  variables  which  must  be  considered  in  every 
seat  release.  Even  experienced  personnel  must  be  careful 
not  to  overlook  any  of  the  decision  factors.  An  expert 
system  could  check  known  variables  against  requirements  and 
consistently  determine  an  accurate  seat  release. 

Expert  Selection 

The  next  step  was  to  select  three  human  experts.  An 
expert  is  defined  as  "one  who  is  probably  better  at 
performing  in  a  domain  than  those  who  are  not  considered  to 
be  expert"  (18:34).  The  domain  experts  must  be,  besides 
experts  in  their  field,  available  and  willing  to  participate 
in  the  study.  Given  these  criteria,  the  three  experts 
selected  for  this  study  were  Master  Sergeant  (MSgt)  James  M. 
McDonald,  Technical  Sergeant  (TSgt)  Stephen  A.  Chatfield  and 
Staff  Sergeant  (SSgt)  James  R.  Ratliff.  According  to  Mr. 
Altick,  these  three  individuals  were  exceptionally  well 
qualified,  available,  and  willing  to  participate  in  the 
study. 

MSgt  McDonald  is  currently  the  Superintendent  of  the 
Air  Passenger  Terminal  at  Wright  Patterson  AFB,  OH.  He  has 
more  than  15  years  experience  in  air  transportation  and  has 
worked  in  air  terminal  operations  for  10  years.  Prior  to 
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his  current  assignment  he  was  the  Noncommissioned  Officer  in 
Charge  (NCOIC)  of  the  ATOC  at  Anderson  AFB,  Guam. 

TSgt  Chatfield  is  currently  the  Superintendent  of  the 
ATOC  at  Wright  Patterson  AFB,  OH.  He  has  more  is  12  years 
experience  in  air  transportation  and  has  worked  in  air 
terminal  operations  for  7  years.  Prior  to  his  current 
assignment,  he  was  a  Shift  Supervisor  in  the  ATOC  at  Clark 
Air  Base  the  Philippines. 

SSgt  Ratliff  is  currently  a  Shift  Chief  in  the  ATOC  at 
Wright  Patterson  AFB.  He  has  more  than  9  years  experience 
in  air  transportation  and  has  worked  in  air  terminal 
operations  for  5  years.  Prior  to  his  current  assignment,  he 
was  a  Senior  Controller  in  the  ATOC  at  Elmendorf  AFB, 

Alaska. 

Selection  of  an  Expert  System  Shell 

VP  Expert,  version  2.1,  by  Paperback  Software 
International  was  selected  as  the  expert  system  shell.  VP 
Expert  is  one  of  the  least  expensive  ($88)  of  all  the 
commercially  available  shells,  yet  has  many  of  the  qualities 
of  more  expensive  software  and  meets  all  requirements  for 
this  project.  The  software  is  widely  available  through 
commercial  software  sources  and  is  also  available  on  many  of 
the  computers  in  AFIT  School  of  Systems  and  Logistics. 

Selection  of  the  Knowledge  Acquisition  Techniques 

Three  knowledge  acquisition  techniques  were  selected 
from  among  the  many  techniques  identified  in  literature. 
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The  techniques  selected  were  the  interview,  task 
observation,  and  concept  mapping.  A  brief  description  of 
each  technique  and  the  reasons  for  selecting  it  are  provided 
below.  A  more  complete  description  of  these  techniques  can 
be  found  in  Appendix  B. 

The  Interview.  Interviewing  for  knowledge 
involves  a  direct  dialogue  between  the  expert  and  the 
knowledge  engineer.  Simply,  the  knowledge  engineer  asks 
questions  and  the  expert  answers  them.  Through  this 
process,  exchange  knowledge  is  transferred. 

The  interviewing  technique  is  the  technique  cited  most 
often  in  the  literature.  Of  19  sources  which  identified 
known  knowledge  acquisition  techniques,  all  identified 
interviewing  as  a  means  of  collecting  knowledge  from  human 
experts.  According  to  Waldron,  "the  interview  remains  the 
primary  knowledge  acquisition  tool"  (31:31).  Given  this 
universal  recognition  of  the  interview  as  a  knowledge 
acquisition  technique,  it  seems  only  proper  to  select  it  as 
one  of  three  to  study. 

Task  Observation .  In  task  observation,  the 
knowledge  engineer  watches  the  expert  solve  a  real  problem 
or  perform  a  task.  Only  after  the  task  is  complete  does  the 
knowledge  engineer  ask  questions  about  what  occurred. 

At  the  beginning  of  the  study,  task  observation 
appeared  to  be  a  technique  that  would  lend  itself  to  the 
problem  area  selected  for  the  study.  Many  of  the  activities 
and  decisions  made  in  the  ATOC  are  repetitive  and  readily 
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observable.  The  ATOC  is  also  a  busy  place  which  functions 
best  when  personnel  are  not  interrupted.  Task  observation 
had  the  potential  to  capture  the  processes  experts  use  as 
they  go  about  dealing  with  recurring  situations  and 
decisions.  The  expert  would  be  allowed  to  work  freely  with 
little,  if  any,  interruption. 

Concept  Mapping .  Concept  mapping  is  a  graphical 
approach  to  knowledge  acquisition.  Real  problems  are 
represented  in  the  form  of  drawings  showing  the  objects  in 
the  problem  area  and  the  connections  between  these  objects. 

Concept  mapping  appeared  to  be  a  good  method  for 
extracting  knowledge  in  a  form  which  would  facilitate 
meeting  the  syntax  requirements  of  the  expert  system  shell, 
VP  Expert.  As  described  in  Appendix  B,  the  concept  map  is 
constructed  of  several  "concept  words"  connected  by  "linking 
words."  VP  Expert  dictates  that  rule  labels  and  variable 
names  contain  no  more  than  40  characters  (30:4.3).  The  fact 
that  the  concepts  are  already  in  an  abbreviated  form  in  the 
concept  map  had  the  potential  to  make  formatting  rules  for 
VP  Expert  easier. 

Knowledge  Acquisition 

The  next  major  step  in  the  research  process  was  to 
perform  knowledge  acquisition  using  each  of  the  three 
techniques.  One  knowledge  acquisition  technique  was 
randomly  matched  with  each  of  the  three  experts.  This  was 
accomplished  by  writing  the  names  of  each  technique  and  each 
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expert  on  3  x  5  cards.  One  technique  card  was  pulled  from  a 
hat  and  matched  with  one  expert  card  also  pulled  from  a  hat. 
The  interview  technique  was  assigned  to  MSgt  McDonald, 
concept  mapping  was  assigned  to  TSgt  Chatfield,  and  the  task 
observation  method  was  assigned  to  SSgt  Ratliff. 

The  Interview  Technique .  The  interviews  were 
conducted  over  five  sessions,  each  lasting  approximately  one 
hour.  Each  session  was  recorded  with  a  tape  recorder. 
Written  notes  were  also  taken.  The  first  22  minutes  of  the 
first  session  were  used  to  get  acquainted  with  the  expert 
and  to  explain  the  purpose  of  the  research.  This  was 
followed  by  an  open-ended  question  concerning  passenger  seat 
release  procedures.  The  purpose  was  to  get  a  general  feel 
for  the  procedures  and  to  find  a  starting  point  from  which 
to  develop  more  specific  questions.  Because  the  researcher 
had  an  idea  of  the  type  of  information  that  was  needed,  the 
remaining  interviews  were  largely  unstructured,  allowing  the 
expert  to  cover  topics  in  his  own  way. 

Shortly  after  each  session,  the  tapes  were  transcribed 
and  reviewed  for  the  purpose  of  preparing  for  the  next 
interview  session. 

The  Task  Observation  Technique .  The  expert  was 
observed  performing  his  tasks  over  a  period  of  five  one  hour 
sessions.  During  the  observation  sessions,  the  expert  was 
permitted  to  perform  his  dut  i.es  uninterrupted.  Only  after 
the  task  was  complete  was  the  expert  questioned  about  any 
part  of  the  task  not  clearly  understood  by  the  researcher. 
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Written  notes  were  taken  throughout  the  observations  and  the 
sessions  were  also  tape  recorded.  During  the  first  session, 
15  minutes  were  used  to  get  acquainted  with  the  expert  and 
to  explain  the  research  project.  The  expert  was  then  asked 
to  explain  the  basic  procedures  for  releasing  seats  to  the 
passenger  terminal.  This  took  another  37  minutes.  The 
remaining  time  in  the  first  session  and  subsequent  sessions 
was  spent  in  the  actual  observation  of  tasks  and  follow  up 
questions . 

The  Concept  Mapping  Technique .  Because  the 
researcher  had  no  previous  experience  using  concept  mapping, 
the  technique  was  practiced  on  persons  unrelated  to  this 
research  using  many  of  the  examples  provided  in  Gowin  and 
Novak's  book,  Learning  How  To  Learn  (23).  The  concept 
mapping  sessions  were  conducted  over  five  sessions,  each 
lasting  approximately  one  hour  each.  Each  session  was  also 
recorded  with  a  tape  recorder.  During  the  first  session,  16 
minutes  were  spent  getting  acquainted  with  the  expert  and 
explaining  the  research  project.  This  was  followed  by  51 
minutes  of  practice  using  the  concept  mapping  technique. 

The  examples  provided  in  Gowan  and  Novak's  book  were  again 
used  for  this  purpose.  Some  ATOC  examples,  not  related  to 
passenger  seat  releases,  were  also  used  during  the  practice. 
Shortly  after  each  session,  the  maps  were  cleaned  up  and 
redrawn  and  in  some  cases  restructured  in  preparation  for 
the  next  session.  Each  session  built  upon  the  map  produced 
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in  the  previous  session  until  a  complete  map  had  been 
produced. 

The  Comparative  Study 

The  results  of  the  knowledge  acquisition  sessions  were 
recorded  and  compared  using  both  quantitative  and 
qualitative  data.  Quantitative  data  included  the  time  taken 
to  prepare  for  each  knowledge  acquisition  session,  the 
actual  time  spent  in  interaction  with  the  expert,  the  time 
spent  transcribing  and  integrating  information  from  the 
tapes,  written  notes  and  drawings,  the  number  of  expert 
system  rules  generated  by  each  knowledge  acquisition 
technique,  the  number  of  translations  necessary  to  convert 
the  original  data  to  expert  system  rules,  and  the  time  to 
formulate  those  rules.  Qualitative  data  included  the 
researcher's  judgment  about  the  advantages  and  disadvantages 
of  using  each  of  the  techniques  as  well  as  the  ease  of  use 
and  data  translation.  The  results  of  the  analysis  are 
presented  in  Chapter  IV. 

Bui lding  the  Prototype  System 

Based  on  the  results  of  the  comparative  analysis  of  the 
knowledge  acquisition  techniques,  one  technique  was  selected 
to  use  in  the  actual  development  of  a  prototype  expert 
system.  This  expert  system,  called  the  "ATOC  Advisor,"  was 
programmed  to  find  the  number  of  seats  that  could  be 
released  to  the  Air  Passenger  Terminal  for  use  by  space 
available  travelers.  In  addition  to  finding  the  correct 
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number  of  seats  to  release,  the  system  was  designed  to  make 
recommendations  on  inflight  meal  orders  for  the  passengers, 
on  passenger  restrictions  and  limitations,  and  on  passenger 
comfort.  Because  of  the  limited  time  allotted  for  the 
actual  programming  of  an  expert  system  and  the  overall 
purpose  of  this  research  effort — evaluating  knowledge 
acquisition  techniques — most  the  programming  effort  was 
spent  on  rule  development  and  inferencing  strategies. 

Verification  and  Val idation 

Verification  and  validation  are  essential  steps  in  the 
development  of  expert  systems.  Verification  refers  to 
building  the  system  right,  while  validation  refers  to 
building  the  right  system  (24:90).  Both  issues  were  used  to 
further  support  the  usefulness  of  the  technique  selected  to 
develop  the  ATOC  Advisor. 

Verification.  Verification  is  accomplished  throughout 
the  encoding  process.  Each  rule  in  the  expert  system  must 
be  checked  for  completeness  and  consistency  (22:70). 
Verification  was  conducted  by  the  researcher  and  involved 
checking  the  transcribed  computer  code  line-by-line  for 
redundant,  conflicting,  or  incomplete  rules.  It  also 
involved  running  the  expert  system  with  the  "rules  and 
values  windows"  of  VP  Expert  in  the  active  mode.  This  mode 
allowed  the  researcher  to  observe  the  execution  of  each  rule 
as  the  inference  engine  worked  through  the  knowledge  base 
and  assigned  values  to  the  variables  involved  in  the 


27 


passenger  seat  release  process.  With  the  expert  system 
verified,  the  next  step  was  to  validate  the  system. 

Validation.  Validation  seeks  to  answer  the  question  of 
just  how  closely  the  system  performs  to  human  expert  levels 
(24:91).  To  validate  the  ATOC  Advisor,  a  test  was  conducted 
in  the  field  with  the  cooperation  of  personnel  assigned  in 
the  ATOC  at  Wright  Patterson  AFB.  Six  seat  release 
scenarios  (Appendix  C)  were  developed  by  the  researcher  and 
other  experienced  transportation  officers.  These  scenarios 
were  presented  to  the  experts,  who  were  asked  to  solve  each 
of  the  scenarios  manually  while  three  other  non-experts  were 
selected  and  asked  to  solve  the  scenarios  using  the  expert 
system.  Discrepancies,  if  any,  between  the  two  sets  of 
solutions  were  evaluated  to  determine  if  they  resulted  from 
errors,  in  encoding  the  system,  faulty  assumptions  made  by 
the  researcher  while  programming  the  system,  or  errors  in 
the  transfer  of  knowledge. 

Summary 

Chapter  III  described  the  methods  that  were  used  to 
evaluate  and  compare  the  effectiveness  of  three  knowledge 
acquisition  techniques  used  to  gather  knowledge  from  an  Air 
Terminal  Operations  Center  in  an  Air  Force  Transportation 
Squadron.  Interview,  concept  mapping,  and  task  observation 
method  were  selected  for  the  study  and  evaluated  based  on 
both  quantitative  and  qualitative  data. 
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A  prototype  expert  system  was  developed  using  VP  Expert 
and  the  knowledge  gathered  from  one  the  three  techniques. 

The  system  was  validated  in  a  field  test  to  further  support 
the  effectiveness  of  the  technique  selected  as  the  more 
effective  among  the  three  studied. 
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IV .  Analysis  of  Results 


Overview 

The  purpose  of  this  chapter  is  to  analyze  the  results 
of  this  study  and  determine  which  of  the  three  knowledge 
acquisition  techniques  compared — interview,  task 
observation,  or  concept  mapping — was  most  useful  in 
developing  an  expert  system  for  the  Air  Terminal  Operations 
Center  (ATOC) .  The  results  are  presented  in  four  main 
parts:  technique  implementation,  rule  production, 
qualitative  judgements,  and  expert  system  validation. 

Technique  Implementation 

The  results  presented  in  this  section  are  concerned 
with  the  actual  execution  of  each  knowledge  acquisition 
technique.  In  addition  to  the  actual  time  spent  interacting 
with  the  experts,  the  time  spent  transcribing  and 
integrating  information  from  written  notes  and  audiotapes, 
as  well  as  the  total  time  involved  in  preparing  for  each 
technique  was  recorded.  Table  1  presents  the  results  of 
this  phase  of  the  evaluation. 

Interaction  Time .  Interaction  time  included  only  the 
actual  time  spent  engaged  in  the  activity  of  knowledge 
transfer.  The  time  taken  to  get  acquainted  with  the 
experts,  to  brief  them  on  the  purpose  of  this  research,  or 
to  practice  a  technique  was  not  included  in  this  time. 
Additionally,  any  interruption  time  for  such  things 
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Table  1 

Knowledge  Acquisition  Technique 


Implementation 

Times  (in 

minutes) 

Interview 

Concept 

Mapping 

Task 

Observation 

Interaction 

Times 

209.9 

192.0 

227.0 

Transcription 

238.5 

367.2 

253.6 

Preparation 

Times 

41.3 

51.0 

18.2 

Total  Time 

489.7 

610.2 

498.8 

as  unrelated  phone  calls  or  personal  breaks  was  also  not 
included  in  the  interaction  times. 

Task  observation  had  the  highest  interaction  time  of 
the  three  techniques.  This  can  be  attributed  to  the  fact 
that  there  was  less  conversation  between  the  researcher  and 
the  expert  and  that  most  activity  in  which  the  expert 
engaged  was  a  target  for  observation  and  considered  as 
interaction  time.  Concept  mapping  had  the  lowest 
interaction  time  because  the  first  of  the  five  one  hour 
sessions  was  primarily  used  to  practice  the  concept  mapping 
technique . 

Transcription  Time .  Transcription  time  consisted  of 
the  time  spent  transcribing  information  from  audiotapes, 
integrating  that  information  with  written  notes,  and 
redrawing  concept  maps.  It  should  be  noted  here,  that  the 
audiotapes  were  not  transcribed  word-for-word.  If  they  had 
been,  transcription  times,  especially  for  the  interviews. 
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would  have  been  considerably  longer.  A  sample  of  a  "word- 
for-word"  transcription  of  10  minutes  of  interview  time  is 
provided  in  Appendix  D.  This  particular  transcript  took  30 
minutes  to  produce. 

As  indicated  in  Table  1,  transcription  time  was  the 
most  time  consuming  component  of  the  three  time  factors 
recorded.  It  accounted  for  60%  of  the  total  concept  mapping 
time,  50%  of  the  interview  time,  and  49%  of  the  task 
observation  time.  Compared  with  the  interview  and  task 
observation  techniques,  concept  mapping  took  about  one  and 
one  half  times  longer  to  transcribe  than  the  other  two. 

This  can  be  attributed  in  part  to  the  requirement  to  clean 
up  and  redraw  the  concept  maps  after  each  session. 

Preparation  Time.  The  interview  preparation  time 
consisted  of  identifying  questions  to  start  each  interview 
session.  Questions  arising  during  the  transcription  of  the 
interview  tapes  were  noted  and  addressed  during  the 
following  interview.  The  preparation  time  for  concept 
mapping  was  spent  teaching  the  expert  about  the  concept 
mapping  technique  and  having  the  expert  generate  some 
unrelated  practice  concept  maps.  Questions  used  during  the 
actual  concept  mapping  sessions  were  generated  as  individual 
maps  were  drawn.  For  the  task  observations,  very  little 
preparation  time  was  required.  Some  questions  resulting 
from  previous  observations  were  developed  and  used  when  the 
results  of  the  next  actual  observations  were  still  not 


32 


clear.  The  time  required  to  prepare  those  particular 
questions  was  minimal,  as  the  results  in  Table  1 
indicate . 

Total  Time.  Overall,  interviewing  took  the  least  time 
to  implement,  followed  by  task  observation  and  then  concept 
mapping.  If  time  were  the  major  factor  to  consider  when 
choosing  among  knowledge  acquisition  techniques,  then  the 
interview  would  appear  to  be  the  more  useful  among  the  three 
compared  in  this  research. 

Rule  Production 

This  part  of  the  analysis  presents  the  results  of  five 
quantitative  measures  used  to  evaluate  the  usefulness  of 
each  knowledge  acquisition  technique  in  producing  rules. 
These  quantitative  measures  were: 

1.  The  time  required  to  formulate  rules; 

2.  The  total  number  of  rules  generated; 

3.  The  number  of  inferences  necessary  to  complete 
rules ; 

4.  The  number  of  gaps  in  the  information  contained  in 
each  knowledge  set;  and 

5.  The  number  of  translations  necessary  to  convert  the 
expert  knowledge  to  reasonably  correct  expert  system  rules. 

Description  of  Rule  Production.  Rule  production  begins 
with  reading  through  the  transcripts  produced  with  each 
knowledge  acquisition  technique  and  identifying  "chunks"  of 
meaning  (3:72-73).  These  chunks  then  need  to  be  sorted  and 
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grouped  according  to  their  relationship  with  other  chunks 
(3:73-74).  Finally,  the  sorted  chunks  of  information  are 
transformed  and  represented  as  production  rules  (3:74). 

A  production  rule  is  a  rule  that  follows  an  "If-Then" 
format,  but  is  in  a  narrative  form  that  does  not  necessarily 
match  the  syntax  required  for  the  expert  system  software. 

For  example: 


IF:  Hazardous  cargo  has  been  scheduled  to  move  on 
the  aircraft  AND  the  hazardous  cargo  has  been 
designated  "cargo  aircraft  only"  AND  no  waiver  of 
hazardous  restriction  has  been  obtained  from 
higher  headquarters 

THEN:  No  seats  can  be  released 


This  same  rule  would  then  need  to  be  converted  to  a  syntax 
acceptable  for  encoding  in  the  VP  Expert  Shell,  such  as: 

IF  Hazard  =  yes 

AND  CargoOnly  =  yes 
AND  Waiver  =  no 
OR  Waiver  =  unknown 

THEN  Release  =  0 

Rule  Formulation  Time.  The  first  measure  of  a 
technique's  usefulness  in  rule  formulation  consisted  of  the 
time  that  it  took  to  formulate  three  production  rules  from 
each  of  the  three  knowledge  acquisition  data  sets.  For  this 
measure,  the  task  of  releasing  seats  to  the  passenger 
terminal  was  broken  down  into  four  categories  representing 
the  types  of  aircraft  which  offer  seats  for  space  available 
passengers.  The  first  category  consisted  of  releasing  seats 
on  a  C-5  aircraft;  the  second,  releasing  seats  on  a  C-141 
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aircraft;  the  third,  releasing  seats  on  a  C-130;  and  the 
fourth,  releasing  seats  on  "all  other"  types  of  aircraft. 

The  process  of  releasing  seats  is  similar  in  all  four 
categories,  but  each  category  has  some  differences  from  the 
other  categories.  Using  the  last  number  in  each  aircraft's 
designator  and  the  number  9  for  "all  other"  aircraft,  a 
random  number  generator  was  used  to  randomly  select  one  of 
the  four  categories.  Then,  three  production  rules  were 
generated  for  the  selected  category  using  the  knowledge  data 
set  collected  from  the  interviewing  technique.  After  the 
three  rules  had  been  generated  for  the  interview  data  set, 
the  process  was  repeated.  Again,  one  of  the  four  categories 
was  randomly  selected  and  three  more  production  rules  were 
generated  using  the  knowledge  from  the  concept  mapping 
technique.  Finally,  the  same  procedure  was  used  to  generate 
three  production  rules  using  the  knowledge  acquired  from 
task  observations. 

The  procedure  of  randomly  assigning  a  different 
category  of  the  seat  release  process  to  each  knowledge 
acquisition  technique  was  necessary,  so  that  the  researcher 
was  forced  to  look  for  new  "chunks"  of  knowledge  when 
formulating  rules  from  each  knowledge  acquisition  technique 
and  was  not  influenced  by  what  had  been  learned  from 
producing  a  previous  set  of  rules.  These  selection  and 
assignment  procedures  for  rule  formulation  are  illustrated 
in  Figure  3.  The  times  required  to  formulate  each 
production  rule  are  presented  in  Table  2. 
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Concept  mapping  required  the  least  time  to  formulate 
production  rules  for  an  expert  system.  Concept  mapping  also 
demonstrated  the  least  variation  in  time  for  the  first. 
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Rule 

Table  2 
Formulation 
(in  minutes) 

Times 

Rule 

Interview 

Concept 

Mapping 

Task 

Observation 

1 

9.72 

3.12 

6.07 

2 

3.60 

2.08 

4.98 

3 

2.44 

2.36 

3.53 

Total 

15.76 

7.56 

14.58 

Average 

5.25 

2.52 

4.86 

second,  and  third  rules  among  the  three  techniques.  The 
higher  times  for  the  first  rules  in  each  of  the  techniques 
were  caused  by  having  to  search  through  the  transcripts  to 
find  related  chunks  of  information.  The  times  are  much 
higher  for  the  interview  and  task  observation  techniques 
because  of  the  way  the  material  was  organized  after 
transcription.  It  was  easier  to  locate  the  needed 
information  on  the  concept  map  than  iji  the  pages  of  the 
interview  and  task  observation  transcripts.  While  it 
appears  that  the  subsequent  formulation  times  approach  those 
of  the  concept  map,  the  unorganized  data  from  the  interviews 
could  be  expected  to  increase  required  time  as  more  complex 
rules  were  formulated.  Additionally,  the  initial  search 
time  to  identify  related  data  could  be  expected  each  time  a 
different  portion  of  the  seat  release  process  was  attempted. 
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Total  Rules  Generated  and  Inferences  Required.  This 


measure  consisted  of  1),  the  number  of  rules  which  could  be 
generated  with  each  technique  given  a  particular  category  of 
the  seat  release  procedure  and  2),  the  number  of  inferences 
which  were  required  to  complete  each  rule.  Inferences  are 
required  whenever  the  knowledge  engineer  cannot  find  an 
explicit  statement  of  relationship  in  the  data  and  thus  must 
establish  the  relationship  by  inferential  reasoning. 
Knowledge  engineers  seek  to  minimize  the  number  of 
inferences  they  draw. 

Again,  the  task  of  releasing  seats  to  the  passenger 
terminal  was  broken  down  into  the  four  categories  described 
earlier:  releasing  seats  on  a  C-5  aircraft,  releasing  seats 
on  a  C-14I  aircraft,  releasing  seats  on  a  C-130,  and 
releasiig  seats  on  "all  other"  types  of  aircraft.  Using  the 
last  number  in  each  aircraft's  designator  and  the  number  9 
for  "all  other"  aircraft,  a  random  number  generator  was  used 
to  select  one  of  the  four  categories.  Then,  using  one 
knowledge  set  after  another,  all  possible  rules  were 
generated  for  the  selected  category.  The  result  was  three 
complete  sets  of  rules — each  set  derived  from  a  different 
acquisition  technique.  As  the  rules  were  formulated,  the 
number  of  inferences  required  to  complete  a  rule  were  noted. 

The  procedures  for  measuring  the  number  of  rules 
generated  from  each  of  the  knowledge  acquisition  techniques 
are  illustrated  in  Figure  4.  The  results  of  these  measures 
are  presented  in  Table  3. 
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Figure  4.  Procedure  for  Measuring  Number  of  Rules 
Formulated  and  Inferences  Required. 


Table  3 

Number  of  Rules  Formulated 
and  Inferences  Required  to  Formulate  Rules 


Interview 


Concept 

Mapping 


Task 

Observation 


Number  of 
Rules 

Number  of 
Inferences 
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Concept  mapping  provided  two  more  rules  than  the 
interview  technique  and  seven  more  rules  than  the  task 
observation  technique.  The  information  in  the  concept  map 
which  generated  the  additional  rules  was  missing  from  the 
transcripts  of  the  interview  and  the  task  observations. 
Although  additional  interviews  or  more  time  spent  in  the 
observation  of  tasks  might  have  supplied  this  missing 
information,  given  the  fixed,  and  somewhat  limited  time 
allotted  for  each  technique,  concept  mapping  was  more  useful 
for  rule  generation,  both  in  the  larger  number  of  rules  it 
supported  and  the  smaller  number  of  inferences  required  in 
the  rules. 

Concept  mapping  also  required  the  fewest  inferences  to 
complete  its  set  of  rules.  Task  observation  required  the 
greatest  number  of  inferences  because  the  researcher  had  to 
assume  the  rationale  for  many  of  the  actions  the  expert  took 
in  performing  his  tasks.  An  explanation  was  provided  for 
the  actions  that  the  expert  took  only  when  the  researcher 
clearly  did  not  understand  the  action  being  observed  and 
asked  for  clarification.  As  a  consequence,  the  knowledge 
gained  by  this  technique  was  inferred  from  the  expert's 
actions  rather  than  verbally  explained.  The  results  from 
this  test  reflect  the  fact  that  the  knowledge  engineer  made 
assumptions  about  rules  for  instances  when  he  did  not  ask 
the  expert  the  exact  reason  for  the  expert's  action. 

Gaps  in  the  Knowledge  Base .  Gaps  in  the  knowledge  base 
were  areas  identified  in  the  transcripts  of  the  interviews 
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and  task  observations  or  in  the  concept  map  where  the 
information  needed  to  produce  expert  system  rules  was 
missing.  The  number  of  gaps  in  a  knowledge  base  is  an 
indicator  of  the  incompleteness  of  that  knowledge  base. 

For  this  test,  a  general  diagram  of  the  seat  release 
procedure  was  developed  using  the  knowledge  collected  during 
the  course  of  all  knowledge  acquisition  sessions.  This 
diagram  was  not  based  on  the  knowledge  collected  from  any 
one  technique,  but  represented  the  combined  knowledge 
acquired  by  the  researcher  using  all  three  acquisition 
techniques  during  the  course  of  this  study. 

Next,  the  knowledge  collected  from  each  of  the  three 
acquisition  techniques  used  in  this  research  was  compared  to 
this  diagram.  If  a  particular  step  in  the  seat  release 
procedure  was  missing  from  the  knowledge  data  set  obtained 
with  a  particular  technique,  then  an  'x'  was  placed  next  to 
this  step  in  the  diagram  under  the  column  of  the  applicable 
knowledge  acquisition  technique.  This  'x'  represented  a 
gap  in  the  knowledge  collected  by  that  particular  technique. 
The  general  diagram  of  the  seat  release  process  is  presented 
in  Figure  5.  The  total  number  of  gaps  found  for  each 
technique  is  located  at  the  bottom  of  the  last  diagram  on 
page  44. 

The  results  indicate  that  the  concept  mapping  technique 
made  the  greatest  contribution  to  the  total  knowledge  base 
since  there  was  only  one  area  of  knowledge  that  it  did  not 
capture  from  the  expert. 
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Passenger  Seat  lelease  is  based  on 


Interview 
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- ^Standard  timber  of  Seats 
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Task 
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— »88  for  Two  Loadaasters 


Figure  5.  General  Diagram  of  the  Seat  Release  Process 
Indicating  Gaps  in  Knowledge  (continues) 
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Passenger  Seat  Release  is  based  on 
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Figure  5  (continued).  General  Diagram  of  the  Seat  Release 
Process  Indicating  Gaps  in  Knowledge  (continues) 
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Figure  5  (continued).  General  Diagram  of  the  Seat  Release 
Process  Indicating  Gaps  in  Knowledge 


Number  of  Knowledge  Transformations .  This  part  of  the 
analysis  refers  to  the  number  of  transformations  the 
knowledge  must  undergo  to  become  an  expert  system  rule  with 
proper  syntax.  Table  4  provides  the  results  of  this  test. 


Table  4 

Number  of  Knowledge  Transformations 
Necessary  to  Create  an  Expert  System  Rule 


Concept  Task 

Interview  Mapping  Observation 


Required 

Transformations  3  23 


44 


The  steps  involved  in  preparing  an  expert  system  rule 
from  the  interview  consisted  of  transcribing  the  information 
contained  on  audiotape  to  writing,  producing  a  production 
rule  from  the  written  text,  and  then  formulating  expert 
system  rules  in  the  syntax  required  by  the  development 
shell,  VP  Expert. 

The  first  transformation  required  with  concept  mapping 
was  to  redraw,  reorganize,  and  integrate  the  rough  concept 
maps  produced  in  each  individual  session  and  produce  one 
complete  concept  map  of  the  entire  seat  release  process. 
Because  the  graphical  representation  of  knowledge  in  the 
concept  map  provided  clear  and  concise  relationships  among 
the  data,  expert  system  rules  were  formulated  directly  from 
the  concept  map,  eliminating  the  step  needed  for  the 
development  of  production  rules. 

Task  observation  involved  the  same  steps  as  the 
interview.  Information  was  transcribed  from  audiotape  and 
integrated  with  notes  to  produce  a  written  text.  Production 
rules  were  produced  from  the  written  text  and  then 
transformed  into  expert  system  rules. 

Summary  of  Rule  Production .  Concept  mapping  produced 
more  rules,  in  less  time,  and  with  fewer  inferences  than  the 
interview  or  task  observation  techniques.  Concept  mapping 
also  produced  the  most  complete  knowledge  base.  The 
interview  was  the  simplest  technique  to  implement  in  the 
researcher's  judgement,  while  the  concept  map  presented  the 
knowledge  in  a  form  which  was  the  easiest  for  the  researcher 
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to  use  when  formulating  rules.  As  a  result,  the  concept  map 
took  one  less  transformation  than  the  interview  or  task 
observation  techniques  to  arrive  at  an  expert  system  rule. 

Qual itative  Judgements 

The  quantitative  measurements  of  the  three  knowledge 
acquisition  techniques  do  not  reflect  all  the  relevant 
information  with  regard  to  using  each  of  the  techniques. 
There  are  a  couple  of  qualitative  judgements  which  should  be 
considered.  They  are  the  ease  with  which  each  technique  was 
implemented  and  the  ease  in  transforming  the  knowledge  into 
production  and  expert  system  rules. 

Ease  of  Technique  Implementation.  This  researcher  felt 
that  the  interviewing  technique  was  the  easiest  method  to 
use  during  the  knowledge  acquisition  sessions.  The  question 
and  answer  format  for  knowledge  transfer  was  easily 
understood  by  both  the  researcher  and  the  expert.  The 
interview  required  little  preparation  time,  and  it  was 
relatively  easy  to  transcribe  the  audiotapes  of  the 
interviews  to  writing. 

Ease  of  Knowledge  Transformation .  Once  the  knowledge 
had  been  acquired,  the  graphical  representation  of  knowledge 
as  provided  by  the  concept  mapping  technique  proved  the 
easiest  to  use  when  producing  expert  system  rules.  Not  only 
were  the  relationships  clear  and  concise,  but  a  concept  map 
required  the  least  amount  of  time  to  search  and  find  all  of 
the  pertinent  relationships.  Both  the  interview  and  the 
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task  observation  methods  were  difficult  to  work  with  when  it 


came  to  sifting  through  the  pages  of  written  transcripts  for 
the  purpose  of  producing  production  rules. 

The  most  difficult  knowledge  to  work  with  was  that 
collected  by  task  observation.  This  data  was  loosely 
structured  and  incomplete.  Because  all  of  the  possible 
situations  that  could  have  an  effect  on  a  seat  release  did 
not  occur  during  the  time  allotted  to  make  observations, 
large  gaps  were  present  in  the  knowledge  base. 

Expert  System  Development  and  Val idation. 

This  part  of  the  analysis  pertains  to  the  selection  and 
further  evaluation  of  the  concept  mapping  technique  which 
was  used  to  build  a  prototype  expert  system  for  determining 
passenger  seat  releases. 

Technique  Selection.  In  terms  of  the  time  required  to 
implement  each  of  the  knowledge  acquisition  techniques,  the 
interview  was  more  useful  among  the  three  techniques 
compared  in  this  study.  However,  in  terms  of  the 
completeness  of  the  knowledge  base  and  the  ability  to 
produce  more  rules,  in  less  time,  and  with  fewer  inferences, 
concept  mapping  was  the  most  useful  technique.  Because  the 
time  had  already  been  invested  in  implementing  all  three 
techniques,  it  was  not  a  factor  in  the  selection  of  a 
technique  to  use  for  further  expert  system  development. 
Therefore,  base  on  its  ability  to  produce  rules,  concept 
mapping  was  selected  as  the  technique  to  use  to  develop  a 
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prototype  seat  release  expert  system.  The  complete  concept 
map  used  to  program  the  expert  system  can  be  found  at 
Appendix  E. 

Expert  System  Development .  The  coding  of  the  expert 
system  was  accomplished  by  the  researcher  using  the  text 
editor  provided  with  the  VP  Expert  Shell.  The  completed 
prototype  expert  system  contained  1443  lines  of  code  and  123 
rules.  An  example  of  the  program  is  provided  at  Appendix  F. 
The  complete  program  is  contained  on  a  5.25  inch  diskette 
which  can  be  obtained  from  AFIT/LSC,  Wright-Patterson  AFB, 
Ohio  45431. 

Expert  System  Validation.  The  expert  system  was 
validated  in  a  field  test  according  to  the  procedures 
identified  in  the  methodology  chapter.  The  purpose  of  this 
validation  was  twofold.  First,  the  purpose  was  to  determine 
if  the  knowledge  base  of  the  expert  system  was  complete  in 
the  judgement  of  the  expert  who  contributed  to  it  through 
the  concept  mapping  technique.  Second,  the  purpose  was  to 
determine  if  the  results  produced  by  the  expert  system 
matched  the  results  produced  by  all  three  experts. 

For  the  first  part  of  the  validation,  the  contributing 
expert  worked  through  several  of  his  own  seat  release 
scenarios  using  the  expert  system.  In  each  case  he  found 
that  the  expert  system  produced  the  same  solution  that  he 
would  have  arrived  at  had  he  determined  the  seat  release 
manually.  In  his  judgement  the  knowledge  base  contained  in 
the  expert  system  was  complete  and  accurate. 
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For  the  second  part  of  the  validation,  three  non¬ 
experts  were  asked  to  solve  six  seat  release  scenarios 
according  to  the  instructions  provided  with  the  scenarios 
(see  Appendix  C).  The  three  experts  used  in  this  study  were 
asked  to  solve  the  same  six  scenarios  using  manual  methods. 
The  manual  solutions  provided  by  the  experts  for  each 
scenario  were  then  compared  to  solutions  found  by  non¬ 
experts  using  the  expert  system.  In  all  cases,  the 
solutions  arrived  at  by  the  non-experts  matched  those  of  the 
experts  and  were,  in  fact,  the  correct  solutions. 
Additionally,  it  took  less  time  to  solve  the  scenarios  using 
the  expert  system  than  it  did  using  manual  methods.  The 
times  required  to  solve  each  scenario  is  presented  Table  5. 


Table  5 

Seat  Release  Scenario  Solution  Times 
(Time  in  Minutes) 


Scenario 

Manual 

Solution  Times 
12  3 

Expert  System 
Solution  Times 
12  3 

1 

3.7 

4.6 

4.0 

2.6 

3.2 

2.7 

2 

1.9 

1.4 

1.8 

1.4 

1.0 

0.8 

3 

3.5 

4.0 

3.7 

2.2 

2.4 

1.7 

4 

3.8 

3.6 

4.3 

2.5 

2.9 

2.1 

5 

4.1 

4.2 

3.6 

2.0 

2.3 

1.8 

6 

1.2 

0.9 

0.9 

1.3 

0.8 

0.7 

Average  Time 

3.0 

3.1 

3.0 

2.0 

2.1 

1.6 

49 


The  results  of  this  field  test  support  the  validity  of 
the  expert  system  and  the  completeness  of  the  knowledge  base 
it  contains.  Additionally,  the  field  test  shows  the 
advantage  of  using  an  expert  system  in  terms  of  time  taken 
to  arrive  at  a  seat  release. 

Summary 

Chapter  IV  presented  the  results  of  the  comparative 
study  of  the  three  knowledge  acquisition  techniques.  The 
interview  technique  proved  more  useful  when  the  time 
required  to  obtain  a  transcript  of  the  knowledge  was  of 
primary  concern  to  the  knowledge  engineer.  However,  concept 
mapping  was  more  useful  in  producing  more  rules  in  less  time 
than  the  other  two  techniques.  Concept  mapping  was  also 
easier  to  use  when  transforming  the  acquired  knowledge  into 
expert' system  rules.  The  ability  of  the  concept  mapping 
technique  to  produce  a  complete  and  correct  knowledge  base 
for  an  expert  system  was  validated  in  a  field  test  of  the 
expert  system. 
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V.  Summary ,  Conclusions,  and  Recommendations 


Overview 

The  purpose  of  this  chapter  is  to  review  the  research 
effort,  present  the  conclusions  drawn  from  the  analysis,  and 
suggest  recommendations  for  future  research. 

Summary  of  the  Research  Effort 

Despite  continued  advances  in  expert  system  technology, 
the  process  of  acquiring  the  expert  knowledge  that  will  make 
up  the  most  important  component  of  an  expert  system — the 
knowledge  base — had  not  been  comparatively  studied.  It  is  a 
problem  that  most  limits  the  application  of  expert  system 
technology.  Many  acquisition  techniques  have  been 
identified  in  the  literature,  but  very  little  data  is 
available  on  which  technique,  or  combination  of  techniques, 
may  be  more  useful  in  different  situations. 

This  research  effort  compared  three  knowledge 
acquisition  techniques — interviewing,  concept  mapping,  and 
task  observation — used  to  gather  knowledge  from  an  Air 
Terminal  Operation  Center.  The  objective  was  to  determine 
which  of  these  three  techniques  was  more  useful  in  acquiring 
the  knowledge  needed  to  develop  an  expert  system.  This 
expert  system  was  designed  to  determine  the  number  of  seats 
to  release  to  the  passenger  terminal  for  use  by  space 
available  passengers.  The  comparison  was  based  on  the  ease 
with  which  each  technique  was  implemented,  the  ability  of 
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each  technique  to  produce  a  complete  and  accurate  knowledge 
base,  and  the  ability  of  each  technique  to  produce  knowledge 
in  a  form  that  simplified  the  programming  effort. 

Concept  mapping  proved  to  be  the  more  useful  technique 
based  on  its  ability  to  produce  more  rules,  in  less  time, 
and  with  fewer  inferences.  It  also  produced  the  knowledge 
in  a  form  that  required  one  less  conversion  prior  to  the 
encoding  process.  As  further  support  of  its  usefulness,  a 
prototype  expert  system  was  developed  using  the  concept  map. 
This  system  was  then  validated  in  a  field  test. 

Conclusions 

This  section  provides  the  answers  and  conclusions  to 
the  research  questions  presented  in  Chapter  I. 

Conclusion  to  Research  Question  1..  The  first  research 
question  asked  what  are  the  current  and  most  widely 
recognized  techniques  used  to  acquire  knowledge  from 
experts.  A  review  of  the  literature  on  expert  system 
technology  and  knowledge  acquisition  revealed  that  there  are 
several  methods  of  knowledge  acquisition.  These  methods  can 
be  classified  into  two  categories:  those  that  attempt  to 
acquire  knowledge  that  can  be  explicitely  stated  or  observed 
and  those  that  attempt  to  acquire  knowledge  that  must  be 
inferred  by  indirect  methods. 

Of  the  techniques  that  focus  on  explicit  knowledge,  the 
interview  is,  by  far,  the  most  common  and  widely  used  of 
these  knowledge  acquisition  techniques.  Some  other 
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techniques  in  this  category  that  are  discussed  frequently  in 
the  literature  on  knowledge  acquisition  are  task 
observation,  questionnaires,  and  protocol  analysis.  Also 
mentioned,  but  much  less  frequently,  are  interruption 
analysis,  inferential  flow  analysis  and  drawing  closed 
curves . 

The  most  common  and  widely  recognized  technique  used  to 
acquire  knowledge  by  inference  is  repertory  grid  analysis. 
The  literature  also  discusses  concept  mapping, 
multidimensional  scaling,  hierarchical  clustering,  and 
general  weighted  networks  as  methods  that  can  be  used  to  get 
at  the  knowledge  that  experts  cannot  easily  verbalize. 

Conclusion  to  Research  Question  2.  The  second  research 
question  asked  what  current  Air  Force  issue  in  the  ATOC  was 
appropriate  for  building  a  prototype  expert  system  and  what 
software  development  tool  could  be  used  to  build  it.  For 
this  research  effort,  passenger  operations  was  selected  as 
the  problem  area  to  address  with  a  prototype  expert  system. 
VP  Expert,  an  expert  system  shell  from  Paperback  Software 
International,  was  selected  as  the  expert  system  development 
tool . 

Within  the  area  of  passenger  operations,  the  process  of 
determining  the  number  of  seats  to  release  for  use  by  space 
available  travelers  was  appropriate  for  an  expert  system 
application.  Air  Force  managers  have  a  high  degree  of 
interest  in  this  process  because  space  available  travel  is 
publicized  as  a  "service  benefit"  for  recruiting  and 
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retention  purposes.  However,  the  procedures  for 
determining  seat  releases  are  unstructured  and  require  the 
consideration  of  a  number  of  different  variables.  Further, 
personnel  qualified  and  experienced  with  the  procedures  are 
frequently  lost  due  to  assignment  changes,  separation  and 
retirement.  In  fact,  of  the  three  experts  used  in  this 
study,  one  will  retire  in  December  of  1990,  and  another  will 
be  moving  to  another  base  in  September  1990. 

A  prototype  expert  system  was  successfully  developed 
using  VP  Expert.  The  software  is  able  to  run  on  all 
hardware  available  in  the  Wright  Patterson  AFB  ATOC.  Most 
important,  three  non-experts,  using  the  expert  system,  were 
able  to  determine  correctly  the  seat  release  in  a  field  test 
of  six  seat  release  scenarios  in  less  time  than  it  took  the 
experts  to  determine  the  releases  manually. 

Conclusion  to  Research  Question  3.  The  third  question 
asked  which  extraction  technique  among  three  evaluated  in 
the  research  was  more  effective  in  producing  the  knowledge 
that  would  be  programmed  into  the  prototype  expert  system. 
The  answer  to  this  question  is  that  it  depends  on  the 
situation  facing  the  knowledge  engineer.  As  a  result,  a 
format  very  similar  to  the  production  rules  presented  in 
Chapter  III  is  used  to  present  the  conclusions  to  this 
research  question. 

If  it  is  not  possible  to  observe  all  possible  events  in 
a  problem  area  being  considered  for  an  expert  system 
application,  then  the  task  observation  method  of  knowledge 
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acquisition  is  not  an  effective  method.  In  this  study,  task 
observation  proved  to  be  very  ineffective,  because  many 
potential  situations  involving  seat  release  procedures  did 
not  occur. 

If  time  available  to  spend  with  the  expert  is  limited 
and  is  the  most  important  factor  to  the  knowledge  engineer 
about  to  begin  an  expert  system  project  and  if  the  knowledge 
engineer  has  little  or  no  prior  experience  with  concept 
mapping  or  if  the  expert  feels  uncomfortable  with  concept 
mapping,  then  interviewing  would  be  the  more  useful 
technique.  The  data  in  this  study  show  that  the  interview 
took  less  time  to  use  than  the  other  two  techniques. 

However,  it  should  be  noted  that  this  researcher  lacked 
experience  with  using  concept  mapping,  a  fact  which 
contributed  to  the  higher  implementation  time  recorded  in 
this  study.  A  knowledge  engineer,  experienced  with  using 
concept  mapping  might  find  it  takes  less  time  to  use  than 
the  other  techniques. 

If  the  time  required  to  transform  raw  knowledge  into 
properly  syntaxed  expert  system  rules  is  the  primary 
consideration,  then  concept  mapping  would  be  the  more  useful 
technique.  In  this  study,  it  took  less  time  to  develop 
rules  from  the  concept  map  that  it  did  from  the  transcripts 
of  the  interviews  and  task  observations.  Additionally,  the 
prototype  expert  system  was  programmed  directly  from  the 
concept  map.  This  eliminated  the  additional  step  of 
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formulating  production  rules  which  was  required  with  the 
interview  and  task  observation  transcripts. 

If  the  completeness  of  the  knowledge  base  is  of  primary 
concern  to  the  knowledge  engineer,  then  a  combination  of 
interviewing  and  concept  mapping  would  be  the  most  effective 
knowledge  acquisition  method.  An  incomplete  knowledge  base 
could  invalidate  the  expert  system.  In  this  study,  there 
were  gaps  in  the  knowledge  data  sets  of  all  three 
techniques;  however,  the  combination  of  the  data  from  the 
interview  and  the  concept  map  produced  a  complete  knowledge 
base . 

Conclusion  to  Research  Question  4.  The  fourth  and 
final  research  question  asked  whether  the  knowledge  base 
that  was  used  to  build  the  expert  system  was  complete  and 
expert  system  valid.  The  knowledge  base  used  in  the 
prototype  expert  system  developed  in  this  research  effort 
was  valid.  In  a  field  test  of  six  seat  release  scenarios  it 
produced  the  same  correct  solutions  as  those  produced  by  the 
experts.  Additionally,  the  content  of  the  knowledge  base 
was  reviewed  by  the  expert  who  had  contributed  to  the 
concept  map  that  was  used  to  develop  the  expert  system.  He 
found  the  knowledge  base  to  be  complete. 

Recommendations  for  Future  Research 

This  thesis  examined  only  three  of  many  recognized 
knowledge  acquisition  methods.  Further  research  in  this 
area  would  be  useful  in  providing  additional  information  on 
the  effectiveness  of  other  knowledge  acquisition  methods 
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used  for  developing  experts  systems.  This  research  cou’d 
take  many  different  directions.  One  possibility  would  be  to 
select  three  different  techniques  for  comparison.  Another 
possibility  would  be  to  compare  one  or  more  techniques  in 
combination  with  other  techniques.  For  example,  the 
interview  technique  combined  with  task  observation  could  be 
compared  to  the  interview  combined  with  concept  mapping. 
Still  another  possibility  would  be  to  compare  the  three 
techniques  used  in  this  research  using  a  different  problem 
situation  and  a  different  development  tool.  It  is  obvious 
that  the  possible  combinations  of  study  are  numerous.  The 
important  point  is  that  continued  research  in  this  area  will 
provide  additional  information  in  an  area  where  information 
is  hard  to  find. 

Another  important  recommendation  for  further  research 
is  in  the  application  of  expert  system  technology  in  the  Air 
Terminal  Operations  Center  (ATOC).  The  expert  system 
developed  for  this  thesis  was  a  by-product  of  the 
comparative  study  of  knowledge  acquisition  methods  and  was 
small  by  comparison  to  any  expert  system  that  might  have 
been  developed  had  that  been  the  topic  of  this  thesis. 
However,  the  system  did  prove  successful  in  providing  an 
expert  solution  to  a  problem  in  less  time  than  the  manual 
methods  used  by  the  real  experts.  The  Air  Force  would 
benefit  from  a  research  effort  that  applied  expert  system 
technology  to  the  ATOC  environment  or  similar  transportation 
activities . 
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Appendix  A: 


Definitions  of  Terms  used  in  the  Report 
and  the  Expert  System 


Additional  Crew  Member  (ACM)  -  An  individual  with  valid 
aeronautical  orders  who  is  required  to  perform  inflight 
duties  and  is  assigned  in  addition  to  or  authorized  to 
accompany  the  normal  aircrew  compliment  required  for  a 
mission.  (12:5) 

Aerial  Port  -  An  airfield  which  has  been  designated  for 
sustained  air  movement  of  traffic  and  to  serve  as  an 
authorized  port  of  entrance  or  departure  to  or  from  the 
country  in  which  located.  (12:5) 

A1 lowable  Cabin  Load  -  The  total  load  an  aircraft  can 
transport  over  a  given  distance  taking  into  account  weight 
and  volume.  (12:5) 

Artificial  Intel  1 iqence  -  The  branch  of  computer  science 
devoted  to  the  study  of  how  computers  can  be  used  to 
simulate  and  duplicate  the  process  by  which  humans  solve 
problems.  (21:1) 

Comfort  Pallet  -  A  self  contained  unit  having  two  lavatories 
and  a  galley  for  storing  and  preparing  inflight  meals 
which  can  be  loaded  on  aircraft  to  support  additional 
passengers.  (12:7) 

Conclusion  -  The  "THEN"  part,  or  consequent,  of  a  rule  in  VP 
Expert.  (30:2-4) 

Domain  -  The  application  area  for  which  an  expert  system  is 
being  developed.  (21:8) 

Goal  -  The  results  or  achievement  toward  which  effort  is 
directed.  In  VP  Expert,  a  goal  is  an  expression  for  which  a 
value  is  being  sought.  (30:2-2) 

Hazardous  Cargo  -  Any  material  which  (by  virtue  of  its 
properties)  is  flammable,  corrosive,  an  oxidizing  agent, 
explosive,  compressed  gas,  poisonous,  an  irritating 
agent,  radioactive,  magnetic,  and  items  not  otherwise 
specified.  (12:11) 

Inlap  -  Infants  less  than  two  years  old  who  may  travel  in 
the  lap  of  an  adult  sponsor  at  no  charge.  (12:11) 
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Knowledge  Engineer  -  The  individual  who  is  responsible  for 
knowledge  acquisition,  representation,  and  programming 
phases  of  developing  an  expert  system.  (1:20) 

Limiting  Wing  Fuel  Weight  -  That  weight  expressed  in  pounds 
where  an  addition  to  the  aircraft  gross  weight  can  be  made 
only  by  adding  fuel  in  the  wing  tanks.  Value  is  also 
referred  to  as  the  Zero  Fuel  Weight.  (12:12) 

Loadmaster  -  An  aircrew  member  who  accomplishes  loading  and 
offloading  aircraft  functions,  insures  the  safety  and 
security  of  cargo  and  baggage  in  flight,  and  provides  for 
the  safety  and  comfort  of  passengers  in  flight.  (12:12) 

MAC  Mission  Observer  (MMO)  -  Personnel  who  have  been  invited 
by  the  Commander-in-Chief  Military  Airlift  Command  (CINCMAC) 
to  accompany  MAC  aircrews  on  MAC  military  aircraft.  (12:12) 

Maximum  Takeoff  Weight  -  Stated  allowable  upper  limit,  in 
pounds,  for  a  particular  aircraft  while  at  take  off.  (12:13) 

Maximum  Ramp  Weight  -  Stated  allowable  upper  limit,  in 
pounds,  for  a  particular  aircraft  while  sitting  on  the 
airfield.  (12:13) 

Mission  Essential  Ground  Personnel  (MEGP)  -  Individuals 
who  perform  unique  support  duties  directly  related  and 
essential  to  a  particular  aircraft,  aircrew  or  numbered 
mission.  (12:13) 

Operating  Weight  -  Basic  aircraft  weight  plus  the  weight  of 
crew  members,  crew  baggage,  oil,  emergency  equipment, 
stewards  equipment,  and  extra  equipment.  (12:14) 

Premise  -  The  "IF"  part,  or  antecedent,  of  an  rule  in  VP 
Expert.  (30:2-3) 

Rule  -  A  regulation  or  statement  defining  a  particular 
conduct,  habit  or  behavior.  In  an  expert  system,  the  rule 
states  that  if  a  given  premise  is  true,  then  a  specific 
action  should  be  taken.  (30:2-3) 
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Appendix  B: 

Description  of  Knowledge  Acquisition  Techniques 

Interviewing 

The  most  common  method  of  knowledge  acquisition  is  the 
face-to-face  interview.  Through  conversation,  experts  are 
asked  to  verbalize  how  they  go  about  solving  a  problem 
(25:153).  In  its  simplest  form  the  interviewer  asks 
questions  to  which  the  interviewee  responds  with  answers. 

The  answers  are  collected,  most  often  with  the  aid  of  a  tape 
recorder,  and  subsequently  transcribed,  analyzed,  and  coded 
(31:31)  . 

The  interview  can  be  structured  or  unstructured.  In  a 
structured  interview,  the  same  questions  are  asked  in  the 
same  order  with  the  same  words  for  each  interview.  The 
questions  are  usually  closed,  in  that  the  expected  responses 
will  be  short  and  to  the  point.  The  structured  interview  is 
useful  when  specific  material  is  required  and  the  responses 
can  be  anticipated.  The  unstructured  interview  allows  the 
interviewees  to  cover  topics  in  their  own  way,  usually  in 
response  to  an  opening  question  followed  by  additional 
probing  questions  or  prompts.  The  unstructured  interview  is 
useful  when  greater  understanding  of  the  structure  of  the 
problem  is  required  before  more  structured  questions  can  be 
asked  (10:107-116). 

AI  researchers  have  found  that  the  interview  is  one  of 
the  most  important  tools  for  facilitating  the  transfer  of 
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human  knowledge  (10:107-108;  31:31).  Most  knowledge 
engineering  sessions  begin  with  at  least  an  initial 
interview  to  get  acquainted  with  the  expert  and  to  get  a 
feel  for  the  basic  structure  of  the  problem  domain  (5:229). 
It  is  taken  as  a  starting  point  for  considering  more  formal 
techniques  to  use  in  later  knowledge  acquisition  sessions. 

The  single  biggest  advantage  of  using  the  interviewing 
technique  is  that  it  is  a  natural  process,  which  is  easily 
understood  by  both  the  knowledge  engineer  and  the  expert 
(5:229;  25:153).  However,  interviewing  is  often  more  than 
just  simply  sitting  down  and  talking  with  an  expert.  The 
interview  relies  on  the  expert's  ability  to  articulate  the 
information  used  to  work  through  a  task.  Unfortunately, 
experts  often  have  a  difficult  time  verbalizing  how  they  go 
about  solving  problems  (31:31).  As  a  result,  the  interview 
is  not  always  a  reliable  way  to  obtain  complete,  objective, 
or  well-organized  descriptions  of  complex  cognitive 
processes . 

Table  6  offers  some  general  suggestions  for  the 
interviewer  to  follow  during  the  interview  process. 

Task  Observation 

Task  observation  involves  observing  experts  work  at  a 
real  problem  to  determine  how  they  make  a  decision  (25:153). 
Through  the  direct  observation  of  the  expert  working  a 
problem,  the  knowledge  engineer  has  the  opportunity  to 


61 


Table  6 

General  Suggestions  for  the  Interviewer  (29:204) 


1.  Do  not  allow  the  interview  to  remain  at  the  general 
or  abstract  level  for  too  long.  Get  to  specifics 
in  order  to  general  useful  information. 

2.  Do  not  ask  the  experts  to  represent  their  thinking 
in  ways  that  are  not  familiar  to  them. 

3.  Do  not  interrupt  the  expert's  train  of  thought. 

Be  patient  and  do  not  rush  the  expert,  even  though 
the  expert  seems  to  ramble  on. 

4.  Record  information  carefully.  Using  a  tape 
recorder  is  highly  recommended. 

5.  Be  attuned  to  the  way  in  which  experts  apply  their 
knowledge.  The  explicit  content  of  the  interview 
may  mask  other  important  information  that  is 
implicit  and  less  obvious. 

6.  Remain  flexible.  Avoid  drawing  conclusions  too 
early  and  do  not  be  disappointed  if  a  favorite 
theory  is  proven  incorrect. 


discover  the  objects,  relationships  and  the  inferences  that 
the  expert  uses  when  solving  problems  (15:21).  In  task 
observation,  the  observer  will  watch  the  behavior  or 
activities  of  the  expert  as  they  proceed  naturally,  and  then 
ask  questions.  Observers  often  notice  things  that  the 
experts  take  for  granted  or  are  unaware  of  (15:21).  Like 
the  interview,  task  observations  can  be  structured  or 
unstructured,  though  here  structure  is  in  the  decisions 
regarding  what  to  look  for,  instead  of  what  to  ask  (15:21). 
Like  most  other  acquisition  methods,  task  observation  is  not 
suited  to  achieving  all  knowledge  acquisition  goals,  and 
there  are  limitations  to  the  method  (25:155).  Access  to  the 
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people  and  places  to  be  observed  is  foremost  among  the 
problems  (15:21).  The  problem  may  also  take  a  considerable 
amount  of  time  to  solve.  The  task  observation  method  is  not 
the  method  designed  to  probe  issues  deeply,  but  rather  to 
let  the  situation  speak  for  itself  (15:22;  25:55). 

Concept  Mapping 

Concept  mapping  gives  the  expert  the  opportunity  to 
visualize  real  problems  in  terms  of  individual  objects  in  a 
problem  area  (20:276;  25:159).  It  helps  the  expert  depict 
"relationships  between  concepts  in  the  form  of  propositions" 
(23:15).  Gowin  and  Novak  define  propositions  as  "two  or 
more  concept  labels  linked  by  words  in  a  semantic  unit" 
(23:15).  A  simple  example  of  a  concept  map  forming  a  valid 
proposition  would  be  "sky  is  blue"  with  "sky"  and  "blue"  as 
the  concepts  and  "is"  the  link  between  them  (23:15).  This 
is  drawn  as: 


blue 


Figure  6  gives  a  larger  example  of  a  concept  map  for 
living  things  and  closely  related  concepts. 

Although  infrequently  listed  in  the  knowledge 
acquisition  literature  as  a  technique,  concept  mapping  may 
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living  things 


can  be 


can  be 


contain 


made  of 


changes 


increased  by 


(^~heat^) 


Figure  6.  Concept  Map  for  Living  Things  (23:16) 

prove  to  be  particularly  useful  in  that  the  process  itself 
establishes  limits  on  a  problem  space  and  defines  the 
problem  explicitly.  Concept  maps  provide  structure  to  an 
unstructured  situation. 
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Appendix  C: 

Passenger  Seat  Release  Scenarios  used  in  System  Val idation 


? 

General  Instructions 


1.  Insert  the  5.25  inch  disk,  labeled  "ATOC  Advisor"  in 
Drive  A.  At  the  A>  prompt  type:  "RUN". 

2.  In  a  few  moments  (required  to  load  the  VP  Expert 
Program)  you  will  see  a  short  introductory  screen.  Press 
any  key  to  proceed  with  the  consultation. 

3.  A  simulated  schedule  of  missions  will  then  be  displayed 
on  the  screen.  It  will  ask  you  to  select  one  of  the 
missions  on  the  schedule  or  to  enter  "NEW"  for  any  mission 
not  on  the  schedule.  Select  the  mission  number 
corresponding  to  the  mission  identified  in  Seat  Release 
Scenario  #1. 

4.  From  this  point  on  the  system  will  ask  you  a  series  of 
questions  about  the  selected  mission.  Use  the  scenario 
narrative  to  answer  each  of  these  questions.  If  at  any 
point,  the  answer  is  not  provided  in  the  narrative,  enter 
"?"  for  unknown.  If  you  are  unable  to  proceed  through  the 
consultation  for  lack  of  information,  consult  with  the 
Knowledge  Engineer.  [See  Knowledge  Engineer’s  notes  at  the 
end  of  this  appendix.] 


Seat  Release  Scenarios 


SEAT  RELEASE  SCENARIO  #  1: 

This  aircraft  is  a  C-141,  flying  mission  number 
AQB0707A0065 .  The  estimated  flying  time  is  4  hours.  The 
aircraft  is  configured  CP-3.  The  crew  consists  of  7  basic 
crew  members,  4  ACMs  and  4  MEGPs .  This  crew  compliment 
includes  2  loadmasters.  The  operating  weight  of  the 
aircraft  is  153,630  pounds.  The  planned  fuel  load  is 
135,000  pounds.  The  planned  cargo  load  is  14,600  pounds  of 
cargo,  of  which  1,620  pounds  is  hazardous.  Some  of  the 
hazardous  material  is  "passenger  prohibitive."  There  are  no 
human  remains. 


65 


SEAT  RELEASE  SCENARIO  #  2: 

This  aircraft  is  a  C-5,  flying  mission  number  ABA02F200065 . 
The  estimated  flying  time  is  8  hours.  The  aircraft  is 
configured  CP-3.  The  crew  consists  of  12  basic  crew  members 
and  1  MMO.  This  crew  compliment  includes  3  loadmasters. 

The  operating  weight  of  the  aircraft  is  374,558  pounds.  The 
planned  fuel  load  is  172,000  pounds.  The  plan  cargo  load  is 
68,452  pounds  of  cargo,  none  of  which  is  hazardous.  This 
cargo  load  includes  1232  pounds  for  two  human  remains. 


SEAT  RELEASE  SCENARIO  #  3: 

This  aircraft  is  a  C-141,  flying  mission  number 
ABR04F200065 .  The  estimated  flying  time  is  6  hours.  The 
aircraft  configuration  is  modified  with  a  comfort  pallet  and 
51  aft  facing  seats.  The  crew  consists  of  6  basic  crew 
members  including  2  loadmasters.  The  operating  weight  of 
the  aircraft  is  152,385  pounds.  The  planned  fuel  load  is 
138,000  pounds.  The  planned  cargo  load  is  100,000  pounds  of 
cargo  none  of  which  is  hazardous.  There  are  no  human 
remains . 


SEAT  RELEASE  SCENARIO  #  4: 

This  aircraft  is  a  C-130E,  flying  mission  number 
ABA09C901065 .  The  estimated  flying  time  is  2  hours.  The 
aircraft  is  configured  CP-5.  The  crew  consists  of  5  basic 
crew  members  and  4  MEGPs.  This  crew  compliment  includes  2 
loadmasters.  The  operating  weight  of  the  aircraft  is  83,546 
pounds.  The  planned  fuel  load  is  43,400  pounds.  There  is 
2000  pounds  of  cargo  planned  for  this  mission  including  one 
human  remains  weighing  542  pounds.  There  is  no  hazardous 
cargo . 


SEAT  RELEASE  SCENARIO  #  5: 

This  aircraft  is  a  C-141,  flying  mission  number 
ABA04330A065 .  The  estimated  flying  time  is  8  hours.  The 
aircraft  configuration  is  modified  with  50  troop  seats  and 
no  comfort  pallet.  The  crew  consists  of  7  basic  crew 
members  and  2  ACMs .  This  crew  compliment  includes  2 
loadmasters.  The  operating  weight  of  the  aircraft  is 
151,250  pounds.  The  planned  fuel  load  is  146,000  pounds. 
The  planned  cargo  load  is  13,364  pounds  of  cargo. 

There  are  no  human  remains . 


66 


SEAT  RELEASE  SCENARIO  #  6: 


This  is  a  new  mission,  a  C-12,  mission  number  SWIFT  21.  The 
estimated  flying  time  is  2  hours.  The  schedule  indicates 
that  there  are  2  space  required  passengers  getting  on  this 
station  and  each  is  authorized  40  pounds  of  excess  baggage 
for  a  total  of  80  pounds. 


Knowledge  Engineer 1 s  Notes  (not  provided  to  the  user) 

1.  For  seat  release  scenario  #1,  the  user  should  look  for  a 
"pax  pro"  waiver  number.  The  knowledge  engineer,  playing 
the  role  of  21AF,  will  provide  waiver  number  21AF001  to  the 
user  when  asked. 

2.  For  seat  release  scenario  #3,  there  is  a  error 
concerning  the  cargo  weight.  The  cargo  weight  should  be 
10,000  pounds,  not  100,000  pounds. 

3.  For  seat  release  scenario  #5,  there  is  not  information 
in  the  narrative  about  the  presence  of  hazardous  cargo  in 
the  planned  aircraft  load.  The  knowledge  engineer,  playing 
the  role  of  the  load  planner,  will  apologize  for  forgetting 
to  pass  this  information  along  and  inform  the  user  that 
there  is  450  pounds  of  hazardous  cargo,  but  no  passenger 
prohibitive  cargo. 
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Appendix  D: 

Partial  Transcript  of  an  Interview 


To  illustrate  the  interview  method  of  knowledge 
acquisition,  a  partial  transcript  of  a  sample  discussion 
with  the  expert  is  presented  below.  This  transcript 
describes  the  interaction  between  the  knowledge  engineer 
(KE)  and  the  expert  concerning  seat  release  procedures  for 
aircraft  other  than  the  C-141,  C-5,  and  C-130. 


KE:  On  what  types  of  aircraft  do  you  determine  a  seat 

release  for  use  by  space  available  passengers? 

EXPERT:  Here  at  Wright  Patterson  one  of  the  most  common 
aircraft  we  deal  with  is  the  C-21  aircraft,  a  small 
aircraft,  where  the  seat  releases  a  based  on  six  passengers 
on  an  airplane.  We  get  a  schedule  from  MAC  which  comes  out 
daily  with  the  names  on  it  and  the  itineraries,  call  signs 
for  the  airplanes,  the  times  and  everything.  We  type  the 
manifest  up  from  the  message  we  get  from  MAC,  we  post  the 
times,  and  what  ever  is  left  over  we  open  up  seats.  If  we 
have  six  passengers  we  say  no  seats.  If  we  have  five 
passengers  we  put  down  one  seat  on  the  board  and  go  from 
there.  On  the  larger  airplanes,  we  have  four  types  of  team 
travel..  We  have  the  135  aircraft  which  come  in  from  Offutt. 
They  carry  89  passengers.  Then  we  have  the  boxer  call  sign 
which  are  stationed  at  Andrews,  run  by  the  Air  National 
Guard.  They  have  727s  and  they  have  89  seats.  Then  we  have 
the  Bobcat  missions  which  are  run  out  of  Buckley.  They’re 
the  T-43's  run  by  the  Air  National  Guard  and  they  lave  64 
seats.  They  all  also  come  out  on  a  schedule,  on  a  team 
travel  schedule.  So  we  all  know  what  the  seat  capacities 
are  for  the  airplanes.  For  example,  this  morning  we  had 
this  boxer  come  in  and  I  knew  it  had  89  seats  on  the 
airplane  and  the  schedule  said  it  had  15  people  getting  off 
here.  So  automatically  we  would  release  15  seats  out  to  the 
next  stop.  Then  based  on  the  schedule  we  know  how  many 
people  are  getting  on  and  off  at  the  next  stop  and  the  stop 
after  that  and  so  we  can  release  more  seats  depending  on 
what  the  schedule  is. 

KE:  So,  it's  a  matter  of  taking  the  number  of  seats 

available  on  the  aircraft  and  then  subtracting  the  number  of 
schedule  passengers  to  arrive  at  a  seat  release  for  space 
available  passengers? 

EXPERT:  No,  not  always.  If  we  have  two  people  trying  to  go 

for  one  seat  and  depending  on  what  type  of  plane  it  is,  say 
if  it’s  a  C-12,  we'll  really  go  after  that  and  try  to  get 
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another  seat  because  C-12s  have  a  jump  seat  and  can  really 
take  seven  passengers  if  they  need  to.  When  the  plane  comes 
in,  we'll  check  with  the  crew  and  ask  how  many  seats  we  can 
get  out.  We'll  look  at  the  schedule  and  he  may  say,  "well 
I'm  picking  up  five  people  and  I'll  give  you  one  seat"  or 
"I'll  go  ahead  and  give  you  two  seats".  It  depends  on  where 
there  going.  And  then  the  weather,  you  know.  If  it's 
weather  then  sometimes  they'll  only  give  you  three  seats 
even  though  you  know  the  plane  is  leaving  here  with  no 
scheduled  passengers.  It's  because  they're  putting  extra 
gas  on.  Then  we  look  out  for  excess  baggage.  A  lot  of 
times  the  schedule  will  have  the  name  and  next  to  it  will 
say  how  much  baggage  is  excess,  because  on  the  C-12  and  the 
C-21  the  baggage  is  normally  30  pounds  and  they  know  that 
and  that's  all  they  usually  bring.  So  we  look  at  that,  and 
see  how  many  bags  they  are  authorized  extra.  Like  the 
couriers  a  lot  of  times  are  authorized  a  lot.  Sometimes 
they'll  come  with  200  or  300  pound  boxes.  So  if  there  is 
any  excess  we  always  take  a  seat  away,  maybe  two,  it  depends 
on  the  weight  and  the  types  of  bags.  Usually  if  it's  less 
than  80  pounds  we'll  take  one  seat  away  and  if  it's  more 
than  80  pounds  we'll  take  two  seats  away,  but  again  it  all 
depends  on  the  type  and  size  of  the  excess  baggage  also. 

KE:  Are  you  saying  there  is  no  established  procedure  for 
reducing  seats  because  of  excess  baggage? 

EXPERT:  Yes,  it’s  kind  of  something  you  learn  with 
experience.  You  have  to  look  at  the  bags  or  the  boxes  in 
addition  to  weighing  them  to  get  an  idea  of  how  many  seats 
they'll  take  up  on  the  airplane.  And  its  not  only  true  for 
the  C-12s  and  the  C-21s.  Its  also  true  in  a  way  for  the 
727's.  Normally  a  passenger  is  allowed  to  carry  66  pounds 
of  baggage,  but  on  the  727  it's  45  pounds  when  you  use  all 
the  seats.  That's  because  if  all  89  people  took  66  pounds 
the  aircraft  would  be  too  heavy.  But  that  doesn't  usually 
happen  hear  so  we  don't  worry  about  it  too  much,  but  it  is 
something  you  have  to  think  about. 

KE:  Would  you  once  again  run  down  the  list  of  aircraft  you 
deal  with  most  often  and  the  number  of  seats  each  offer? 

EXPERT:  Okay,  we  start  off  with  the  C-21  and  the  C-12.  The 

capacity  of  both  airplanes  is  six  seats.  We  run  daily 
flights  of  those  aircraft  and  those  are  run  by  MAC.  We  also 
have  the  T-43,  C-22,  and  the  C-135  and  those  are  all  MAC 
team  travel  airplanes.  Now,  the  135  is  a  74  seat  airplane, 
the  C-22  is  an  89  seat  airplane,  the  T-43  is  a  64  seat 
airplane . 
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KE:  And  excess  baggage  applies  only  to  the  smaller  C-12  and 
C-21? 

EXPERT:  Yes,  unless  it's  the  C-22,  that's  the  727,  iF  going 

to  be  full  of  89  passengers.  The  C-22,  the  T-43,  and  the 
C-135  all  have  areas  on  the  airplane  to  stow  baggage,  so 
excess  baggage  is  not  really  a  problem. 

KE:  Are  there  any  other  aircraft  that  you  work  with? 

EXPERT:  Yes.  There  is  the  medivac  C-9  which  we  get  three 

to  four  days  a  week.  Here  the  seats  are  released  by  the 
crew  about  30  minutes  prior  to  leaving  and  they're  based  on 
mission  availability.  We  also  get  some  SAC  aircraft  like 
the  KC-10  and  KC-135.  But  there  again  the  seats  are 
released  by  the  crew  and  we  don't  have  to  much  to  say  about 
them. 


KE:  When  you  say  "released  by  the  crew",  is  there  a 

particular  crew  member  who  has  the  authority  to  release 
seats? 

EXPERT:  Kind  of.  For  the  C-9  we  get  the  seats  from  the 

medical  attendant  and  for  the  KC-10  and  KC-135  we  talk  to 
one  of  the  boom  operators. 

KE:  Okay,  how  about  the  aircraft  I  see  parked  on  the  other 

side  of  the  flightline?  I  see  a  couple  of  the  old  A  model 
C-141s  and  the  EC-135.  Do  you  ever  get  seats  on  those 
aircraft? 

EXPERT:  Seldom.  That's  the  4950th  stuff.  They  pretty  much 

handle  things  for  themselves  and  they  call  us.  We  never 
call  them. 
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Appendix  F: 

Examp 1 e  of  Expert  System  Program 


Following  is  an  example  of  the  expert  system  program. 

It  contains  approximately  one  fourth  of  the  total  program 
code  that  was  developed  during  this  study.  This  particular 
part  of  the  program  determines  the  number  of  passenger  seats 
that  can  be  released  on  a  C141  aircraft. 

EXECUTE ; 

ENDOFF; 

RUNTIME; 

BKCOLOR  =  3; 

ACTIONS 

LOADFACTS  Factfile 
CLS 

WOPEN  1,1,1,20,77,3 

ACTIVE  1 

DISPLAY 

"  PASSENGER  SEAT  RELEASE  RECORD 


MISSION  NUMBER: {16Mission}  ACFT  TYPE: {6Acft} 
FLYING  TIME:  {3FlyTime} 

CONFIGURATION: {6Config}  CREW  SIZE:{4Crew> 

NUMBER  OF  LOADMASTERS : {3Loadmaster> 

OPERATING  WEIGHT: {80p_Wt>  FUEL  WEIGHT : {8Fuel_Wt> 
ACL: {8Acl} 

CARGO  WEIGHT: {7Cargo_Wt}  HAZARDOUS : {4Hazard> 

PAX  PRO: {4PaxPro}  WAIVER  NUMBER : {SWaiver} 

SEAT  RELEASE: 


REMARKS : " 

WOPEN  2,16,2,4,75,7 
ACTIVE  2 
FIND  Mission 
FIND  FlyTime 
RESET  Release 
FIND  Release 
CLS 

FORMAT  Release, 3.0 

DISPLAY  "  You  can  release{3release}  seats." 
DISPLAY  "  " 
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DISPLAY  "  Press  any  key  to  continue..."" 

CLS 

WCLOSE  1 

WCLOSE  2 

FIND  ModConfig 

WOPEN  1,1,1,20,77,3 

ACTIVE  1 

DISPLAY 

"  PASSENGER  SEAT  RELEASE  RECORD 


MISSION  NUMBER: {16Mission}  ACFT  TYPE:{6Acft> 
FLYING  TIME:  {3FlyTime} 

CONFIGURATION:  {Config} {Mod}  CREW  SIZE:{4Crew> 
NUMBER  OF  LOADMASTERS : {3Loadmaster} 

OPERATING  WEIGHT: {80p_Wt>  FUEL  WEIGHT : {8Fue lJWt} 
ACL: {8Acl} 

CARGO  WEIGHT: {7Cargo_Wt}  HAZARDOUS : {4Hazard> 

PAX  PRO: {4PaxPro}  WAIVER  NUMBER: {SWaiver} 

SEAT  RELEASE: {4Re lease} 


REMARKS : " 

WOPEN  3,15,2,5,75,7 

ACTIVE  3 

CLS 

FIND  HRRemark 
RESET  Continue 
FIND  Continue 
CLS; 

!  Rules  Block 
RULE  1 

IF  Config  =  Cl 

THEN  SeatType  =  AftFacing 

CP  ■  No 

Seat_Count=l 1 ;  * 

RULE  2 

IF  Config  =  CPI  * 

THEN  Seat_Type  *  AftFacing 

CP  =  Yes 
Seat  Count=65; 
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RULE 

IF 

THEN 


RULE 

IF 

THEN 


RULE 

IF 

THEN 


RULE 

IF 

THEN 


RULE 

IF 


THEN 


RULE 

IF 


THEN 


RULE 

IF 

THEN 


RULE 

IF 


THEN 


3 

Config  =  CP2 
SeatType  =  AftFacing 
CP  =  Yes 
Seat_Count=  101; 

4 

Config  =  CP3 
SeatType  =  Troop 
CP  =  Yes 
Seat_Count=88 ; 

5 

Config  =  P4 
SeatType  =  Troop 
CP  =  Yes 
S  ea t_C  ount = 170; 

6 

Config  =  P5 
SeatType  =  Troop 
CP  =  No 

Seat_Count=208 ; 

7 

Config  =  Unknown  AND 
Seats  >  0  AND 
SeatType  *  Troop 
SeatCount  =  ((Seats) -2) 

Mod  =  Modified 
Find  CP; 

8 

Config  =  Unknown  AND 
Seats  >  0  AND 
SeatType  =  AftFacing 
SeatCount  =  ((Seats) -4) 

Mod  =  Modified 
Find  CP; 

9 

SeatCount  =  0 
Release  s  NO 
CLS; 

10 

SeatType  =  AftFacing 
AND  Crew  >  13 

AND  SeatCount  <  ( (Crew)-(SeatCount) ) 

Release  ^  NO 

CLS; 


79 


RULE  11 

IF  SeatType  =  Troop 

AND  Crew  >11 

AND  SeatCount  <  ( (Crew)-(Seat_Count) ) 

THEN  Release  =  NO 

CLS; 

RULE  12 

IF  Loadmaster  =  0 

THEN  Release  =  NO 

CLS ; 

RULE  13 

IF  SeatCount  >  0 

AND  Op_Wt  >  0 
AND  FuelWt  >  0 
AND  CargoWt  <=  100000 

THEN  Ac  1  =  ( 325000- ( Op_Wt )- (Fue l_Wt )- (CargoWt ) ) 

FIND  HR 
CLS; 

RULE  14 

IF  Ac  1  <  0 

THEN  Release  =  NO 

CLS 

WOPEN  4,16,2,4,75,4 
ACTIVE  4 

DISPLAY  "WARNING!  You  have  overloaded  this 
aircraft.  Look  at  reducing  fuel  or  cargo  weight. 

Press  any  key  to  continue  .  .  . 

CLS 

ACTIVE  2 
RESET  Ac  1 
RESET  CargoWt 
RESET  FuelWt 
RESET  Release 
FIND  Fuel_Wt 
FIND  Cargo_Wt 
FIND  Acl 
FIND  Release; 

RULE  15 

IF  CargoWt  >  0 

AND  Hazard  *  Yes 
AND  PaxPro  =  Yes 
AND  Waiver  *  NA 
OR  Waiver  =  UNKNOWN 
THEN  Release  =  NO 

CLS 

WOPEN  4,16,2,4,75,4 
ACTIVE  4 

DISPLAY  "WARNING!  -  Some  of  the  hazardous  cargo  you 
are  planning  for  this  mission  is,  as  you  indicated. 
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passenger  prohibitive.  You  must  obtain  a  waiver  from  the  NAF 
before  you  can  release  seats.  <Press  any  key  to  continue>~" 
CLS 

ACTIVE  2 
Reset  Release 
Reset  Waiver 
Find  Release; 

RULE  16 

*  IF  Cargo_Wt  >  0 

AND  Hazard  =  UNKNOWN 
THEN  Release  -  NO 
CLS 

WOPEN  3,16,2,4,75,4 
ACTIVE  3 

DISPLAY  "WARNING!  -  Some  categories  of  hazardous 
cargo  prohibit  the  carrying  of  passengers.  You  must  know 
about  any  hazardous  cargo  to  move  passengers.  Recheck  your 
load  information  and  try  again.  <Press  any  key  to 
continue>~" 

CLS 

ACTIVE  2 
Reset  Hazard 
Reset  Release 
Find  Release; 

RULE  17 

IF  Cargo_Wt  >  0 

AND  Hazard  =  yes 
AND  PaxPro  =  UNKNOWN 
THEN  Release  =  NO 

CLS 

WOPEN  3,16,2,4,75,4 
ACTIVE  3 

DISPLAY  "WARNING!  -  Some  categories  of  hazardous 
cargo  prohibit  the  carrying  of  passengers.  You  must  know 
about  the  type  of  hazard  to  move  passengers.  Recheck  your 
load  information  and  try  again.  <Press  any  key  to 
continued" 

CLS 

ACTIVE  2 
Reset  PaxPro 
Reset  Release 
Find  Release; 

i 

RULE  18 

IF  Ac  1  <=  244 

«  THEN  Release  =  NO; 
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RULE 

IF 

19 

Ac  1  >  244 

THEN 

Weight_Seat  =  ((Acl)/245); 

RULE 

IF 

20 

Seat  Type  =  AftFacing 

AND  Crew  >13 

THEN 

Seat_Crew  =  ( (Seat_Count )- 

RULE 

IF 

21 

Seat  Type  =  Troop 

AND  Crew  >11 

THEN 

SeatCrew  =  ( (Seat_Count )- 

RULE 

IF 

22 

Seat  Type  =  Troop 

AND  Crew  <=11 

THEN 

SeatCrew  =  (SeatCount) ; 

RULE 

IF 

23 

Seat  Type  =  AftFacing 

AND  Crew  <=13 

THEN 

SeatCrew  =  (Seat_Count ) ; 

RULE 

IF 

24 

CP  =  No  AND 

PLimit  <  (SeatCount )  AND 
Weight_Seat  >  (PLimit)  AND 
Seat  Crew  >  (PLimit) 


THEN 

Release  = 

(PLimit) ; 

RULE 

25 

IF 

Loadmaster 

=  1  AND 

Seat  Count 

>  30  AND 

Weight  Seat  >  30  AND 

Seat  Crew 

>  30  AND 

PLimit  >  30 

THEN 

Release  = 

30; 

RULE 

26 

IF 

Loadmaster 

=  2  AND 

Seat  Count 

>  88  AND 

Weight  Seat  >  88  AND 

Seat  Crew 

>  88  AND 

PLimit  >  88 

THEN 

Release  = 

88; 

RULE 

27 

IF 

FLYTIME=1 

THEN 

PLIMIT=( 162 

-(crew) ) ; 

( (Crew) -13) ) ; 


( (Crew)-ll ) ) ; 
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RULE 

IF 

THEN 


RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 

RULE 

IF 

THEN 


28 

FLYTIME=1. 500000 
PLIMIT=( 162- (crew) ) 


29 

FLYTIME=2 

PLIMIT=( 162- (crew) ) 

30 

FLYTIME=2. 500000 
PLIMIT=( 110- (crew) ) 

31 

FLYTIME=3 

PLIMIT=( 108-(crew) ) 

32 

FLYTIME=3. 500000 
PLIMIT=(84-(crew) ) ; 

33 

FLYTIME=4 

PLIMIT=(63-(crew) ) ; 

34 

FLYTIME=4. 500000 
PLIMIT=(56-(crew) ) ; 

35 

FLYTIME=5 

PLIMIT=(51-(crew) ) ; 

36 

FLYTIME=5. 500000 
PLIMIT=(47-(crew) ) ; 

37 

FLYTIME=6 

PLIMIT=(44-(crew) ) ; 

38 

FLYTIME=6. 50000 
PLIMIT=(41-(crew) ) ; 

39 

FLYTIME=7 

PLIMIT=(38-(crew) ) ; 

40 

FLYTIME=7. 500000 
PLIMIT=( 36- (crew) ) ; 


RULE  41 

IF  FLYTIME=8 

THEN  PLIMIT=( 34-(crew) ) ; 


RULE  42 

IF  FLYTIME=8. 500000 

THEN  PLIKIT=(31-(crew) ) ; 

RULE  43 

IF  FLYTIME=9 

THEN  PLIMIT=(31-(crew)) ; 

RULE  44 

IF  FLYTIME=9. 500000 

THEN  PLIMIT=(31-(crew) ) ; 

RULE  45 

IF  FLYTIME=10 

THEN  PLIMIT=(31-(crew) ) ; 

RULE  46 

IF  SeatType  =  Troop  AND 

WeightSeat  >  (Seat_Count)  AND 
Crew  >  11 

THEN  Release  =  ( (Seat_Count )-( (Crew) -11 ) ) ; 

RULE  47 

IF  SeatType  =  AftFacing  AND 

WeightSeat  >  (Seat_Count)  AND 

Crew  >  13 

THEN  Release  =  ( (Seat_Count)-( (Crew) -13) ) ; 

RULE  48 

IF  Weight_Seat  >  (Seat_Count)  AND 

Crew  <=  11  AND 
Seat_Type  =  Troop 
THEN  Release  =  (Seat_Count ) ; 

RULE  49 

IF  WeightSeat  >  (Seat_Count)  AND 

Crew  <=  13  AND 
Seat_Type  =  AftFacing 
THEN  Release  =  (Seat_Count) ; 

RULE  50 

IF  Weight_Seat  <  (Seat_Count) 

THEN  Release  =  (WeightSeat ) ; 
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RULE  51 

IF  HR  =  Yes 

THEN  HRRemark  =  Yes 
DISPLAY 

"Human  remains  (HRs)  are  scheduled  for  this  mission.  If  you 
release  seats  you  must  brief  potential  passengers  that  there 
will  be  HRs  on  the  aircraft  and  that  the  transfer  case(s) 
may  be  close  to  where  they  may  sit. 

<Press  any  key  to  see  next  remark>~" 

CLS ; 

WHENEVER  1 

IF  Continue  =  Yes 

THEN  CLS 

WCLOSE  1 
WCLOSE  2 
CHAIN  Missions 
ELSE  CLS 

DISPLAY 

"Consultation  Ended.  I  hope  you  found  this  program  helpful. 
Any  suggestions  that  you  may  have  to  improve  the  program 
would  be  greatly  appreciated.  Thank  You. 

Press  any  key  to  exit  .  .  . ~ ; 

!  Statements  Block 

ASK  Crew:  "How  many  members  are  there  in  the  aircrew  to 
include  ACMs,  MRSs,  MEGPs ,  and  MMOs? 

RANGE  Crew:  5,30; 

ASK  Seats:  "How  many  seats  are  on  the  aircraft? 

it  • 
i 

RANGE  Seats:  0,208; 

ASK  Loadmaster:  "How  many  loadmasters  are  in  the  crew? 

M  • 
t 

RANGE  Loadmaster:  0,5; 

ASK  Op_Wt:  "What  is  the  operating  weight  of  the  aircraft? 

ti  • 

RANGE  Op_Wt:  130000,165000; 

ASK  FuelWt:  "How  much  fuel  will  be  taken? 

RANGE  Fuel_Wt:  75000,160000; 

ASK  CargoWt:  "How  much  cargo  (in  pounds)  are  you  putting  on 
the  aircraft?  "; 

RANGE  CargoWt :  0,100000; 

ASK  Hazard.  "Is  any  of  this  cargo  hazardous?  (enter  '?'  if 
unknown )  " ; 

CHOICES  Hazard:  Yes, No; 

ASK  PaxPro:  "Is  the  hazardous  cargo  'passenger  prohibitive'? 
(enter  '?'  if  unknown)  "; 

CHOICES  PaxPro:  Yes, No; 

ASK  Waiver:  "What  is  the  PAX  PRO  Waiver  Number? 

(Enter  'NA'  if  not  required  or  '?'  if  unknown.)"; 
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ASK  Continue:  "Would  you  like  to  determine  another  seat 
release?" ; 

CHOICES  Continue:  Yes, No; 

ASK  ConFig:  "How  is  the  aircraft  configured?  (Enter  '?'  if 
Modified) " ; 

CHOICES  ConFig:  CPI , CP2 , Cl , CP3 , P5 , P4; 

ASK  SeatType:  "What  type  seats  are  installed  on  the  a 

aircraft? 

CHOICES  SeatType:  AftFacing, Troop; 

Ask  CP:  "Is  there  a  comfort  pallet  installed?"; 

CHOICES  CP:  Yes, No; 

ASK  FlyTime:  "What  will  be  the  duration  of  the  flight  (in 
hours ) ?  "  ; 

ASK  Mission:  "For  what  mission  do  you  want  to  determine  a 
seat  release?  "; 

ASK  HR:  "Does  this  cargo  include  any  human  remains  (HRs)? 

CHOICES  HR:  Yes, No; 


t 
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